Nov 02 13:56:34.614286 ip-10-0-134-99 systemd[1]: Starting Kubernetes Kubelet... Nov 02 13:56:35.016609 ip-10-0-134-99 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 13:56:35.016609 ip-10-0-134-99 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 13:56:35.016609 ip-10-0-134-99 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 13:56:35.016609 ip-10-0-134-99 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 13:56:35.023224 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023119 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:56:35.023224 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023221 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:56:35.023224 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023227 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:56:35.023224 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023230 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:56:35.023355 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023233 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:56:35.023355 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023237 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:56:35.023355 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023240 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:56:35.023355 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023244 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:56:35.023355 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023247 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:56:35.023355 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023249 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:56:35.023355 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023252 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:56:35.023355 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023255 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:56:35.023355 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023258 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:56:35.023355 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023261 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:56:35.023355 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023263 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:56:35.023355 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023266 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:56:35.023355 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023269 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:56:35.023355 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023271 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:56:35.023355 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023274 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:56:35.023355 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023281 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:56:35.023355 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023284 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:56:35.023355 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023287 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:56:35.023355 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023290 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:56:35.023837 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023293 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:56:35.023837 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023295 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:56:35.023837 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023298 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:56:35.023837 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023301 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:56:35.023837 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023304 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:56:35.023837 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023306 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:56:35.023837 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023309 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:56:35.023837 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023311 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:56:35.023837 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023314 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:56:35.023837 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023316 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:56:35.023837 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023319 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:56:35.023837 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023322 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:56:35.023837 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023325 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:56:35.023837 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023328 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:56:35.023837 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023332 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:56:35.023837 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023335 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:56:35.023837 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023338 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:56:35.023837 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023340 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:56:35.023837 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023343 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:56:35.024304 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023346 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:56:35.024304 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023348 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:56:35.024304 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023351 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:56:35.024304 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023354 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:56:35.024304 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023357 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:56:35.024304 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023360 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:56:35.024304 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023362 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:56:35.024304 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023365 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:56:35.024304 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023370 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:56:35.024304 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023373 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:56:35.024304 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023376 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:56:35.024304 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023378 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:56:35.024304 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023381 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:56:35.024304 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023384 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:56:35.024304 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023387 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:56:35.024304 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023389 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:56:35.024304 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023392 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:56:35.024304 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023395 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:56:35.024304 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023397 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023400 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023403 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023405 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023408 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023410 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023413 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023416 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023419 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023422 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023425 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023428 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023431 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023434 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023437 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023440 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023445 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023449 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023453 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023456 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:56:35.024791 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023458 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:56:35.025291 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023461 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:56:35.025291 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023464 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:56:35.025291 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023467 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:56:35.025291 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023472 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:56:35.025291 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023476 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:56:35.025291 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023479 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:56:35.025291 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023483 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:56:35.025291 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023486 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:56:35.025291 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023488 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:56:35.025291 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023491 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:56:35.025291 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023494 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:56:35.025291 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023496 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:56:35.025291 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023499 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:56:35.025291 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023502 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:56:35.025291 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023505 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:56:35.025291 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023507 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:56:35.025291 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023510 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:56:35.025291 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023513 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:56:35.025291 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023515 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:56:35.025730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023518 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:56:35.025730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023522 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:56:35.025730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023525 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:56:35.025730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023529 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:56:35.025730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023532 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:56:35.025730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.023534 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:56:35.025730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024392 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:56:35.025730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024400 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:56:35.025730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024403 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:56:35.025730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024406 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:56:35.025730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024408 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:56:35.025730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024411 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:56:35.025730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024414 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:56:35.025730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024416 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:56:35.025730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024420 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:56:35.025730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024423 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:56:35.025730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024428 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:56:35.025730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024432 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:56:35.025730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024435 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:56:35.026202 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024438 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:56:35.026202 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024441 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:56:35.026202 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024443 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:56:35.026202 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024446 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:56:35.026202 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024449 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:56:35.026202 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024451 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:56:35.026202 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024454 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:56:35.026202 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024457 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:56:35.026202 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024460 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:56:35.026202 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024463 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:56:35.026202 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024466 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:56:35.026202 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024469 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:56:35.026202 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024472 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:56:35.026202 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024475 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:56:35.026202 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024478 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:56:35.026202 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024481 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:56:35.026202 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024484 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:56:35.026202 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024487 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:56:35.026202 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024491 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024495 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024499 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024502 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024505 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024507 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024510 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024513 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024515 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024523 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024525 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024528 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024531 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024534 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024537 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024539 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024542 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024545 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024548 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024551 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:56:35.026654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024554 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:56:35.027179 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024557 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:56:35.027179 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024559 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:56:35.027179 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024562 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:56:35.027179 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024565 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:56:35.027179 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024567 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:56:35.027179 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024570 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:56:35.027179 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024572 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:56:35.027179 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024575 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:56:35.027179 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024578 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:56:35.027179 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024581 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:56:35.027179 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024584 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:56:35.027179 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024586 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:56:35.027179 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024589 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:56:35.027179 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024592 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:56:35.027179 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024594 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:56:35.027179 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024597 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:56:35.027179 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024599 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:56:35.027179 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024602 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:56:35.027179 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024604 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024607 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024612 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024614 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024617 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024620 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024623 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024626 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024628 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024631 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024634 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024636 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024639 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024642 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024644 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024647 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024650 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024652 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024655 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024658 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:56:35.028789 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024660 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024663 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024666 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024668 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024672 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024675 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024677 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024680 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024684 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024687 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024689 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024692 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024695 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024697 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024701 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.024704 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025232 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025237 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025241 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025244 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:56:35.029401 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025247 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:56:35.029878 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025249 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:56:35.029878 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025252 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:56:35.029878 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025255 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:56:35.029878 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025258 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:56:35.029878 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025260 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:56:35.029878 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025263 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:56:35.029878 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025265 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:56:35.029878 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025268 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:56:35.029878 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025271 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:56:35.029878 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025274 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:56:35.029878 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025276 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:56:35.029878 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025279 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:56:35.029878 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025282 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:56:35.029878 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025285 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:56:35.029878 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025288 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:56:35.029878 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025291 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:56:35.029878 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025294 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:56:35.029878 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025296 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:56:35.029878 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025299 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:56:35.030349 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025304 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:56:35.030349 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025308 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:56:35.030349 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025311 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:56:35.030349 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025314 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:56:35.030349 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025317 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:56:35.030349 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025320 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:56:35.030349 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025325 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:56:35.030349 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025328 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:56:35.030349 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025330 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:56:35.030349 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025333 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:56:35.030349 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025335 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:56:35.030349 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025340 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:56:35.030349 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025342 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:56:35.030349 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025345 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:56:35.030349 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025348 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:56:35.030349 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025351 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:56:35.030349 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025353 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:56:35.030349 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025356 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:56:35.030349 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025359 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:56:35.030852 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025362 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:56:35.030852 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025364 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:56:35.030852 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025367 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:56:35.030852 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025370 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:56:35.030852 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025373 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:56:35.030852 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025376 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:56:35.030852 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025378 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:56:35.030852 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025383 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:56:35.030852 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025386 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:56:35.030852 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025390 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:56:35.030852 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025393 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:56:35.030852 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025395 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:56:35.030852 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025398 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:56:35.030852 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025401 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:56:35.030852 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025404 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:56:35.030852 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025406 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:56:35.030852 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025409 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:56:35.030852 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025412 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:56:35.030852 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025414 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025418 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025421 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025424 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025426 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025429 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025432 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025435 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025438 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025441 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025444 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025447 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025449 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025452 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025454 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025457 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025460 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025462 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025464 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025467 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:56:35.031320 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025470 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025472 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025475 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025477 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025480 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025482 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025485 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025488 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025491 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025494 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025496 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025499 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025502 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025506 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025509 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025513 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025515 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025518 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025520 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025523 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:56:35.031805 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025526 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025529 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025532 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025534 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.025537 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027283 2577 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027295 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027303 2577 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027308 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027312 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027316 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027320 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027325 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027329 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027332 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027336 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027339 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027343 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027346 2577 flags.go:64] FLAG: --cgroup-root="" Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027350 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027353 2577 flags.go:64] FLAG: --client-ca-file="" Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027357 2577 flags.go:64] FLAG: --cloud-provider="external" Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027360 2577 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027364 2577 flags.go:64] FLAG: --cluster-domain="" Nov 02 13:56:35.032287 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027367 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027370 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027374 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027377 2577 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027382 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027385 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027388 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027392 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027395 2577 flags.go:64] FLAG: --contention-profiling="false" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027399 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027402 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027405 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027408 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027413 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027416 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027419 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027422 2577 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027425 2577 flags.go:64] FLAG: --enable-server="true" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027428 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027433 2577 flags.go:64] FLAG: --event-burst="100" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027436 2577 flags.go:64] FLAG: --event-qps="50" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027439 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027442 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027446 2577 flags.go:64] FLAG: --eviction-hard="" Nov 02 13:56:35.032852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027450 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027453 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027456 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027462 2577 flags.go:64] FLAG: --eviction-soft="" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027465 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027468 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027471 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027474 2577 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027477 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027480 2577 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027484 2577 flags.go:64] FLAG: --feature-gates="" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027488 2577 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027492 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027495 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027498 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027502 2577 flags.go:64] FLAG: --healthz-port="10248" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027505 2577 flags.go:64] FLAG: --help="false" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027509 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-134-99.ec2.internal" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027513 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027516 2577 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027519 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027523 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027527 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027530 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 13:56:35.033577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027533 2577 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027536 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027539 2577 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027542 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027545 2577 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027548 2577 flags.go:64] FLAG: --kube-reserved="" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027550 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027553 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027557 2577 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027559 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027562 2577 flags.go:64] FLAG: --lock-file="" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027567 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027571 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027574 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027580 2577 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027583 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027586 2577 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027589 2577 flags.go:64] FLAG: --logging-format="text" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027592 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027596 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027599 2577 flags.go:64] FLAG: --manifest-url="" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027602 2577 flags.go:64] FLAG: --manifest-url-header="" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027607 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027610 2577 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027614 2577 flags.go:64] FLAG: --max-pods="110" Nov 02 13:56:35.034170 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027617 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027620 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027624 2577 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027627 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027630 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027633 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027636 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027644 2577 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027648 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027651 2577 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027654 2577 flags.go:64] FLAG: --pod-cidr="" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027657 2577 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027660 2577 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027663 2577 flags.go:64] FLAG: --pods-per-core="0" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027666 2577 flags.go:64] FLAG: --port="10250" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027669 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027672 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-081e5f5bb815e6c3d" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027675 2577 flags.go:64] FLAG: --qos-reserved="" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027680 2577 flags.go:64] FLAG: --read-only-port="10255" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027683 2577 flags.go:64] FLAG: --register-node="true" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027686 2577 flags.go:64] FLAG: --register-with-taints="" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027690 2577 flags.go:64] FLAG: --registry-burst="10" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027693 2577 flags.go:64] FLAG: --registry-qps="5" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027696 2577 flags.go:64] FLAG: --reserved-cpus="" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027699 2577 flags.go:64] FLAG: --reserved-memory="" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027703 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 13:56:35.034773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027707 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027710 2577 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027713 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027716 2577 flags.go:64] FLAG: --runonce="false" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027719 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027730 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027734 2577 flags.go:64] FLAG: --seccomp-default="false" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027738 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027741 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027745 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027749 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027752 2577 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027755 2577 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027758 2577 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027761 2577 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027764 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027768 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027771 2577 flags.go:64] FLAG: --system-cgroups="" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027774 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027779 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027782 2577 flags.go:64] FLAG: --tls-cert-file="" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027785 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027788 2577 flags.go:64] FLAG: --tls-min-version="" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027791 2577 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027796 2577 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 13:56:35.035413 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027799 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027802 2577 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027805 2577 flags.go:64] FLAG: --v="2" Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027810 2577 flags.go:64] FLAG: --version="false" Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027815 2577 flags.go:64] FLAG: --vmodule="" Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027820 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.027824 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030656 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030676 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030680 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030683 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030686 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030690 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030694 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030696 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030701 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030705 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030709 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030713 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030716 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030719 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:56:35.036032 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030722 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:56:35.036537 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030724 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:56:35.036537 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030727 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:56:35.036537 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030730 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:56:35.036537 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030733 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:56:35.036537 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030735 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:56:35.036537 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030738 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:56:35.036537 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030746 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:56:35.036537 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030750 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:56:35.036537 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030755 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:56:35.036537 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030758 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:56:35.036537 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030761 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:56:35.036537 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030764 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:56:35.036537 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030767 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:56:35.036537 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030770 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:56:35.036537 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030772 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:56:35.036537 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030775 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:56:35.036537 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030778 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:56:35.036537 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030781 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:56:35.036537 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030784 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:56:35.037002 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030788 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:56:35.037002 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030790 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:56:35.037002 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030793 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:56:35.037002 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030796 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:56:35.037002 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030799 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:56:35.037002 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030801 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:56:35.037002 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030804 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:56:35.037002 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030807 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:56:35.037002 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030810 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:56:35.037002 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030813 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:56:35.037002 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030816 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:56:35.037002 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030818 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:56:35.037002 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030821 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:56:35.037002 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030824 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:56:35.037002 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030826 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:56:35.037002 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030829 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:56:35.037002 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030832 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:56:35.037002 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030834 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:56:35.037002 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030838 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030841 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030844 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030846 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030849 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030852 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030855 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030858 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030860 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030863 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030866 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030869 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030872 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030875 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030878 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030880 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030883 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030886 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030888 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030891 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:56:35.037787 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030894 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030897 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030900 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030904 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030908 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030912 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030915 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030918 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030921 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030923 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030926 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030930 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030933 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030936 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030938 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030941 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030957 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030960 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030962 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030965 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:56:35.038637 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030968 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:56:35.039516 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030971 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:56:35.039516 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030974 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:56:35.039516 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030977 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:56:35.039516 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030980 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:56:35.039516 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030983 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:56:35.039516 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030986 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:56:35.039516 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030989 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:56:35.039516 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030992 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:56:35.039516 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030994 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:56:35.039516 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.030997 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:56:35.039516 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031000 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:56:35.039516 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031003 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:56:35.039516 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031005 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:56:35.039516 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.031037 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 13:56:35.039516 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031081 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:56:35.040004 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031086 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:56:35.040004 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031090 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:56:35.040004 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031093 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:56:35.040004 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031096 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:56:35.040004 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031099 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:56:35.040004 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031101 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:56:35.040004 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031104 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:56:35.040004 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031106 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:56:35.040004 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031110 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:56:35.040004 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031112 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:56:35.040004 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031115 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:56:35.040004 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031118 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:56:35.040004 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031120 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:56:35.040004 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031123 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:56:35.040004 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031126 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:56:35.040004 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031128 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:56:35.040004 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031131 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:56:35.040004 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031134 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:56:35.040004 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031137 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:56:35.040472 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031139 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:56:35.040472 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031143 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:56:35.040472 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031146 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:56:35.040472 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031149 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:56:35.040472 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031152 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:56:35.040472 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031154 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:56:35.040472 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031157 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:56:35.040472 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031160 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:56:35.040472 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031163 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:56:35.040472 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031165 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:56:35.040472 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031168 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:56:35.040472 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031171 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:56:35.040472 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031173 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:56:35.040472 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031176 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:56:35.040472 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031179 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:56:35.040472 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031182 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:56:35.040472 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031184 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:56:35.040472 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031187 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:56:35.040472 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031189 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031192 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031195 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031197 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031201 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031203 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031206 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031209 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031212 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031214 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031217 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031219 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031222 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031224 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031227 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031230 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031233 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031236 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031238 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031241 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:56:35.041289 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031243 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031246 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031249 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031251 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031254 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031257 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031261 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031263 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031266 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031270 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031273 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031277 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031279 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031282 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031285 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031288 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031290 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031294 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031296 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031299 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:56:35.042190 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031302 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:56:35.042889 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031304 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:56:35.042889 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031307 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:56:35.042889 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031309 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:56:35.042889 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031312 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:56:35.042889 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031315 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:56:35.042889 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031317 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:56:35.042889 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031320 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:56:35.042889 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031324 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:56:35.042889 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031327 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:56:35.042889 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031330 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:56:35.042889 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031333 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:56:35.042889 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031336 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:56:35.042889 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031339 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:56:35.042889 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031341 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:56:35.042889 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031344 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:56:35.042889 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031347 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:56:35.042889 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031349 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:56:35.042889 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031352 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:56:35.042889 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031355 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031357 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031360 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031363 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031365 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031368 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031370 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031375 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.031377 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.039671 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.039697 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.040438 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.040454 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040551 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040560 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040565 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040570 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040574 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040579 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040583 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:56:35.043580 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040588 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:56:35.044393 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040592 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:56:35.044393 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040599 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:56:35.044393 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040605 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:56:35.044393 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040610 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:56:35.044393 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040615 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:56:35.044393 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040620 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:56:35.044393 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040624 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:56:35.044393 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040628 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:56:35.044393 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040632 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:56:35.044393 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040637 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:56:35.044393 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040641 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:56:35.044393 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040646 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:56:35.044393 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040650 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:56:35.044393 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040654 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:56:35.044393 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040659 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:56:35.044393 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040664 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:56:35.044393 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040668 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:56:35.044393 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040673 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:56:35.044393 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040677 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:56:35.045106 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040681 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:56:35.045106 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040686 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:56:35.045106 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040693 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:56:35.045106 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040697 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:56:35.045106 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040702 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:56:35.045106 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040706 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:56:35.045106 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040711 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:56:35.045106 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040715 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:56:35.045106 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040720 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:56:35.045106 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040724 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:56:35.045106 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040728 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:56:35.045106 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040733 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:56:35.045106 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040737 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:56:35.045106 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040742 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:56:35.045106 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040746 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:56:35.045106 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040751 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:56:35.045106 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040756 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:56:35.045106 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040761 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:56:35.045106 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040765 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040769 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040773 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040777 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040782 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040787 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040791 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040796 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040800 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040804 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040809 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040813 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040817 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040822 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040827 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040831 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040836 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040843 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040847 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040852 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:56:35.045560 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040856 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040860 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040865 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040869 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040874 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040878 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040883 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040887 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040891 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040896 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040900 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040904 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040908 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040913 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040917 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040922 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040927 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040931 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040935 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040940 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:56:35.046215 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040961 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:56:35.046680 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040966 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:56:35.046680 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040970 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:56:35.046680 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040975 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:56:35.046680 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040979 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:56:35.046680 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040984 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:56:35.046680 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040989 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:56:35.046680 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.040993 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:56:35.046680 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041001 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:56:35.046680 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041007 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:56:35.046680 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041014 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:56:35.046680 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041019 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:56:35.046680 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041024 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:56:35.046680 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041028 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:56:35.046680 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041032 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:56:35.046680 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041037 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:56:35.046680 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041041 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:56:35.046680 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041045 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:56:35.046680 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041050 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:56:35.046680 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041054 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:56:35.047273 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041058 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:56:35.047273 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.041105 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 13:56:35.047273 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041343 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:56:35.047273 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041352 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:56:35.047273 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041358 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:56:35.047273 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041362 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:56:35.047273 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041367 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:56:35.047273 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041372 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:56:35.047273 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041376 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:56:35.047273 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041380 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:56:35.047273 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041385 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:56:35.047273 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041389 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:56:35.047273 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041394 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:56:35.047273 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041398 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:56:35.047273 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041403 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:56:35.047273 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041407 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:56:35.047654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041411 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:56:35.047654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041416 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:56:35.047654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041420 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:56:35.047654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041425 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:56:35.047654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041430 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:56:35.047654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041435 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:56:35.047654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041440 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:56:35.047654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041445 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:56:35.047654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041450 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:56:35.047654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041454 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:56:35.047654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041459 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:56:35.047654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041463 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:56:35.047654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041467 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:56:35.047654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041472 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:56:35.047654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041477 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:56:35.047654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041481 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:56:35.047654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041485 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:56:35.047654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041490 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:56:35.047654 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041494 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:56:35.048131 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041498 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:56:35.048131 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041503 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:56:35.048131 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041507 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:56:35.048131 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041511 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:56:35.048131 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041515 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:56:35.048131 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041520 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:56:35.048131 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041524 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:56:35.048131 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041528 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:56:35.048131 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041532 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:56:35.048131 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041537 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:56:35.048131 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041541 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:56:35.048131 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041545 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:56:35.048131 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041550 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:56:35.048131 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041554 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:56:35.048131 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041559 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:56:35.048131 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041563 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:56:35.048131 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041568 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:56:35.048131 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041573 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:56:35.048131 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041577 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:56:35.048585 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041582 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:56:35.048585 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041586 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:56:35.048585 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041592 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:56:35.048585 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041596 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:56:35.048585 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041601 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:56:35.048585 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041605 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:56:35.048585 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041609 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:56:35.048585 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041614 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:56:35.048585 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041618 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:56:35.048585 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041625 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:56:35.048585 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041631 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:56:35.048585 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041636 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:56:35.048585 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041640 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:56:35.048585 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041645 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:56:35.048585 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041649 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:56:35.048585 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041653 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:56:35.048585 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041657 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:56:35.048585 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041662 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:56:35.048585 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041666 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041671 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041675 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041679 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041684 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041688 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041692 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041697 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041701 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041705 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041709 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041714 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041719 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041723 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041728 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041732 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041736 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041741 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041746 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041750 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:56:35.049044 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041754 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:56:35.049524 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041758 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:56:35.049524 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041763 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:56:35.049524 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041767 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:56:35.049524 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041771 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:56:35.049524 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041776 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:56:35.049524 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041780 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:56:35.049524 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041784 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:56:35.049524 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041789 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:56:35.049524 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041793 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:56:35.049524 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041800 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:56:35.049524 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041807 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:56:35.049524 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041812 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:56:35.049524 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041817 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:56:35.049524 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041823 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:56:35.049524 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041828 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:56:35.049524 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.041865 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 13:56:35.049900 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041929 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:56:35.049900 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041937 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:56:35.049900 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041964 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:56:35.049900 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041970 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:56:35.049900 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041974 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:56:35.049900 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041979 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:56:35.049900 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041984 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:56:35.049900 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041989 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:56:35.049900 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041994 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:56:35.049900 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.041999 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:56:35.049900 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042004 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:56:35.049900 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042008 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:56:35.049900 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042012 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:56:35.049900 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042017 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:56:35.049900 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042022 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:56:35.049900 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042027 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:56:35.049900 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042031 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:56:35.049900 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042035 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:56:35.049900 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042040 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042044 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042048 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042052 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042056 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042061 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042065 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042069 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042074 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042078 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042082 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042087 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042091 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042096 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042100 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042104 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042108 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042114 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042120 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042125 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:56:35.050352 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042130 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042136 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042140 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042144 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042148 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042153 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042157 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042161 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042167 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042171 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042175 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042180 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042184 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042188 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042192 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042196 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042200 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042204 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042208 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042212 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:56:35.050823 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042217 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042222 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042228 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042232 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042236 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042241 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042245 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042250 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042254 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042258 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042263 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042267 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042272 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042277 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042281 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042286 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042290 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042294 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042299 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042303 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:56:35.051329 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042307 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:56:35.051795 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042312 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:56:35.051795 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042317 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:56:35.051795 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042321 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:56:35.051795 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042325 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:56:35.051795 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042330 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:56:35.051795 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042334 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:56:35.051795 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042338 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:56:35.051795 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042343 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:56:35.051795 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042347 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:56:35.051795 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042352 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:56:35.051795 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042356 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:56:35.051795 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042361 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:56:35.051795 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042365 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:56:35.051795 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042369 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:56:35.051795 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042374 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:56:35.051795 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042378 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:56:35.051795 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042382 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:56:35.051795 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042386 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:56:35.051795 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042391 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:56:35.052266 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042395 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:56:35.052266 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042399 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:56:35.052266 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042404 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:56:35.052266 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042412 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:56:35.052266 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042421 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:56:35.052266 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042427 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:56:35.052266 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042432 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:56:35.052266 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:35.042436 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:56:35.052266 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.042539 2577 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 13:56:35.052266 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.045744 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 13:56:35.052266 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.046922 2577 server.go:1014] "Starting client certificate rotation" Nov 02 13:56:35.052266 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.047036 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 13:56:35.052266 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.047076 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 13:56:35.052266 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.049397 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 13:56:35.052266 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.052212 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 13:56:35.066517 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.066497 2577 log.go:25] "Validated CRI v1 runtime API" Nov 02 13:56:35.069610 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.069591 2577 log.go:25] "Validated CRI v1 image API" Nov 02 13:56:35.070538 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.070505 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 13:56:35.073309 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.073286 2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 87670cfc-6288-4213-87d4-3acc6ee7e934:/dev/nvme0n1p4 f5e84f0b-4172-44de-a152-373c94270d56:/dev/nvme0n1p3] Nov 02 13:56:35.073392 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.073307 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 13:56:35.080040 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.079181 2577 manager.go:217] Machine: {Timestamp:2026-11-02 13:56:35.077692916 +0000 UTC m=+0.361809098 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106928 MemoryCapacity:33164324864 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b0e2f1ff1f39e3109a5343908f9ad SystemUUID:ec2b0e2f-1ff1-f39e-3109-a5343908f9ad BootID:8ef70e54-d339-4b9e-a488-f8cdfdc46ae1 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cc:14:8d:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cc:14:8d:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:e2:73:e7:7c:4a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164324864 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 13:56:35.080152 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.080027 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 13:56:35.080220 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.080186 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 13:56:35.085224 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.085167 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 13:56:35.085427 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.085228 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-99.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 13:56:35.085494 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.085430 2577 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 13:56:35.085494 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.085440 2577 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 13:56:35.085494 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.085489 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 13:56:35.085577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.085514 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 13:56:35.086880 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.086867 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 13:56:35.091477 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.091456 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 13:56:35.091636 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.091625 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 13:56:35.092463 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.092451 2577 kubelet.go:493] "Attempting to sync node with API server" Nov 02 13:56:35.092512 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.092470 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 13:56:35.092512 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.092487 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 13:56:35.092512 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.092499 2577 kubelet.go:405] "Adding apiserver pod source" Nov 02 13:56:35.092512 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.092511 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 13:56:35.093014 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.093001 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 13:56:35.095680 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.095658 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 13:56:35.103121 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.103097 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 13:56:35.104124 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.104107 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 13:56:35.104205 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.104147 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 13:56:35.104374 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.104361 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 13:56:35.104436 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.104383 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 13:56:35.104436 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.104394 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 13:56:35.104436 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.104405 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 13:56:35.104436 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.104415 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 13:56:35.104436 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.104425 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 13:56:35.105194 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.105183 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 13:56:35.105255 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.105196 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 13:56:35.105255 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.105210 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 13:56:35.105255 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.105219 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 13:56:35.105255 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.105234 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 13:56:35.105255 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.105247 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 13:56:35.106956 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.106928 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 13:56:35.107028 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.106962 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 13:56:35.111736 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.111709 2577 server.go:1263] "Started kubelet" Nov 02 13:56:35.111817 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.111782 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 13:56:35.112141 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.112069 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 13:56:35.112261 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.112243 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 13:56:35.113435 ip-10-0-134-99 systemd[1]: Started Kubernetes Kubelet. Nov 02 13:56:35.114644 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.114585 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 13:56:35.114745 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.114732 2577 server.go:317] "Adding debug handlers to kubelet server" Nov 02 13:56:35.119101 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.119077 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 13:56:35.120474 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.120458 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 13:56:35.120542 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.120459 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 13:56:35.120658 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.120638 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-99.ec2.internal\" not found" Nov 02 13:56:35.121384 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.121353 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 13:56:35.121384 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.121365 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 13:56:35.121384 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.121389 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 13:56:35.121652 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.121631 2577 reconstruct.go:97] "Volume reconstruction finished" Nov 02 13:56:35.121652 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.121650 2577 reconciler.go:29] "Reconciler: start to sync state" Nov 02 13:56:35.121803 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.121789 2577 factory.go:55] Registering systemd factory Nov 02 13:56:35.121847 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.121808 2577 factory.go:223] Registration of the systemd container factory successfully Nov 02 13:56:35.122073 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.122058 2577 factory.go:153] Registering CRI-O factory Nov 02 13:56:35.122130 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.122080 2577 factory.go:223] Registration of the crio container factory successfully Nov 02 13:56:35.122174 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.122141 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 13:56:35.122174 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.122162 2577 factory.go:103] Registering Raw factory Nov 02 13:56:35.122248 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.122203 2577 manager.go:1199] Started watching for new ooms in manager Nov 02 13:56:35.123367 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.123352 2577 manager.go:319] Starting recovery of all containers Nov 02 13:56:35.136869 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.136849 2577 manager.go:324] Recovery completed Nov 02 13:56:35.138330 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.138302 2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 02 13:56:35.140042 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.140011 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 13:56:35.141136 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.141117 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 13:56:35.141607 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.141595 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:56:35.143478 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.143460 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:56:35.143566 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.143498 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:56:35.143566 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.143514 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:56:35.144120 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.144105 2577 cpu_manager.go:226] "Starting" policy="none" Nov 02 13:56:35.144120 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.144118 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 13:56:35.144235 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.144124 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:56:35.144235 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.144134 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 13:56:35.144827 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.144812 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:56:35.144895 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.144841 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:56:35.144895 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.144863 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:56:35.146252 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.146238 2577 policy_none.go:50] "Start" Nov 02 13:56:35.146305 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.146259 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 13:56:35.146305 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.146273 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 13:56:35.148519 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.148507 2577 policy_none.go:44] "Start" Nov 02 13:56:35.159658 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.159633 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 13:56:35.159771 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.159752 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 13:56:35.159987 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.159963 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 13:56:35.161219 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.159973 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-99.ec2.internal.18e43efa02e610c7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-99.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-99.ec2.internal,},FirstTimestamp:2026-11-02 13:56:35.111678151 +0000 UTC m=+0.395794336,LastTimestamp:2026-11-02 13:56:35.111678151 +0000 UTC m=+0.395794336,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-99.ec2.internal,}" Nov 02 13:56:35.175389 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.175362 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mphrc" Nov 02 13:56:35.181729 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.181651 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-99.ec2.internal.18e43efa04cb4cdd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-99.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-99.ec2.internal,},FirstTimestamp:2026-11-02 13:56:35.143478493 +0000 UTC m=+0.427594684,LastTimestamp:2026-11-02 13:56:35.143478493 +0000 UTC m=+0.427594684,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-99.ec2.internal,}" Nov 02 13:56:35.182071 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.182044 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 13:56:35.187688 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.187665 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mphrc" Nov 02 13:56:35.207688 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.193909 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 13:56:35.207688 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.197407 2577 manager.go:341] "Starting Device Plugin manager" Nov 02 13:56:35.207688 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.197434 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 13:56:35.207688 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.197443 2577 server.go:87] "Starting device plugin registration server" Nov 02 13:56:35.207688 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.197668 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 13:56:35.207688 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.197677 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 13:56:35.207688 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.197783 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 13:56:35.207688 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.197878 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 13:56:35.207688 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.197885 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 13:56:35.207688 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.198360 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 13:56:35.207688 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.198415 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-99.ec2.internal\" not found" Nov 02 13:56:35.207688 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.206638 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-99.ec2.internal.18e43efa04cbb6cd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-99.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-99.ec2.internal,},FirstTimestamp:2026-11-02 13:56:35.143505613 +0000 UTC m=+0.427621799,LastTimestamp:2026-11-02 13:56:35.143505613 +0000 UTC m=+0.427621799,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-99.ec2.internal,}" Nov 02 13:56:35.207688 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.207244 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 13:56:35.211132 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.208536 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 13:56:35.211132 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.208558 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 13:56:35.211132 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.208580 2577 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 13:56:35.211132 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.208616 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 13:56:35.232962 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.232927 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:56:35.298624 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.298378 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:56:35.299593 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.299574 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:56:35.299711 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.299608 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:56:35.299711 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.299622 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:56:35.299711 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.299654 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-99.ec2.internal" Nov 02 13:56:35.309164 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.309133 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal"] Nov 02 13:56:35.309265 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.309199 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:56:35.310088 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.310070 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:56:35.310183 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.310106 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:56:35.310183 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.310121 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:56:35.312388 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.312375 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:56:35.312560 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.312544 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" Nov 02 13:56:35.312597 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.312575 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:56:35.313728 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.313713 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:56:35.313816 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.313745 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:56:35.313816 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.313759 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:56:35.313816 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.313713 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:56:35.313984 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.313829 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:56:35.313984 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.313842 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:56:35.316032 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.316018 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" Nov 02 13:56:35.316078 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.316049 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:56:35.316860 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.316841 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:56:35.316931 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.316871 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:56:35.316931 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.316882 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:56:35.321307 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.321292 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-99.ec2.internal" Nov 02 13:56:35.321358 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.321314 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-99.ec2.internal\": node \"ip-10-0-134-99.ec2.internal\" not found" Nov 02 13:56:35.322301 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.322285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/baf43fff2e8eac9bddd0973d4b16dfc0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal\" (UID: \"baf43fff2e8eac9bddd0973d4b16dfc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" Nov 02 13:56:35.322357 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.322309 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/baf43fff2e8eac9bddd0973d4b16dfc0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal\" (UID: \"baf43fff2e8eac9bddd0973d4b16dfc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" Nov 02 13:56:35.322357 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.322326 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcd82cc88301759f1e08e2b014a50651-config\") pod \"kube-apiserver-proxy-ip-10-0-134-99.ec2.internal\" (UID: \"fcd82cc88301759f1e08e2b014a50651\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" Nov 02 13:56:35.352794 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.352773 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-99.ec2.internal\" not found" node="ip-10-0-134-99.ec2.internal" Nov 02 13:56:35.355389 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.355368 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-99.ec2.internal\" not found" node="ip-10-0-134-99.ec2.internal" Nov 02 13:56:35.360408 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.359343 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-99.ec2.internal\" not found" Nov 02 13:56:35.422514 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.422480 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/baf43fff2e8eac9bddd0973d4b16dfc0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal\" (UID: \"baf43fff2e8eac9bddd0973d4b16dfc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" Nov 02 13:56:35.422617 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.422510 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/baf43fff2e8eac9bddd0973d4b16dfc0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal\" (UID: \"baf43fff2e8eac9bddd0973d4b16dfc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" Nov 02 13:56:35.422617 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.422557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/baf43fff2e8eac9bddd0973d4b16dfc0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal\" (UID: \"baf43fff2e8eac9bddd0973d4b16dfc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" Nov 02 13:56:35.422617 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.422578 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcd82cc88301759f1e08e2b014a50651-config\") pod \"kube-apiserver-proxy-ip-10-0-134-99.ec2.internal\" (UID: \"fcd82cc88301759f1e08e2b014a50651\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" Nov 02 13:56:35.422617 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.422610 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcd82cc88301759f1e08e2b014a50651-config\") pod \"kube-apiserver-proxy-ip-10-0-134-99.ec2.internal\" (UID: \"fcd82cc88301759f1e08e2b014a50651\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" Nov 02 13:56:35.422739 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.422640 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/baf43fff2e8eac9bddd0973d4b16dfc0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal\" (UID: \"baf43fff2e8eac9bddd0973d4b16dfc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" Nov 02 13:56:35.459424 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.459394 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-99.ec2.internal\" not found" Nov 02 13:56:35.560118 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.560058 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-99.ec2.internal\" not found" Nov 02 13:56:35.657884 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.657856 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" Nov 02 13:56:35.661001 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.660985 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-99.ec2.internal\" not found" Nov 02 13:56:35.661497 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:35.661482 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" Nov 02 13:56:35.761739 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.761708 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-99.ec2.internal\" not found" Nov 02 13:56:35.862228 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.862161 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-99.ec2.internal\" not found" Nov 02 13:56:35.962718 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:35.962677 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-99.ec2.internal\" not found" Nov 02 13:56:36.046156 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:36.046123 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 13:56:36.046641 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:36.046300 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 13:56:36.063357 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:36.063322 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-99.ec2.internal\" not found" Nov 02 13:56:36.163797 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:36.163770 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-99.ec2.internal\" not found" Nov 02 13:56:36.165417 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:36.165379 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf43fff2e8eac9bddd0973d4b16dfc0.slice/crio-a50b491b0757cf499544e39ae2a906f296f884e34c2263e402a1659550dc730d WatchSource:0}: Error finding container a50b491b0757cf499544e39ae2a906f296f884e34c2263e402a1659550dc730d: Status 404 returned error can't find the container with id a50b491b0757cf499544e39ae2a906f296f884e34c2263e402a1659550dc730d Nov 02 13:56:36.165675 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:36.165652 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd82cc88301759f1e08e2b014a50651.slice/crio-cb03afa4c51dff071823c8c1d7f6ff6f927247e4ee5f9d1a4132d0dcb227b90c WatchSource:0}: Error finding container cb03afa4c51dff071823c8c1d7f6ff6f927247e4ee5f9d1a4132d0dcb227b90c: Status 404 returned error can't find the container with id cb03afa4c51dff071823c8c1d7f6ff6f927247e4ee5f9d1a4132d0dcb227b90c Nov 02 13:56:36.172182 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:36.172165 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 13:56:36.190069 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:36.190048 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:56:36.196682 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:36.196641 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 13:51:34 +0000 UTC" deadline="2028-06-19 07:02:45.09836114 +0000 UTC" Nov 02 13:56:36.196682 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:36.196681 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14273h6m8.90168209s" Nov 02 13:56:36.205765 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:36.205747 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:56:36.211964 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:36.211912 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" event={"ID":"baf43fff2e8eac9bddd0973d4b16dfc0","Type":"ContainerStarted","Data":"a50b491b0757cf499544e39ae2a906f296f884e34c2263e402a1659550dc730d"} Nov 02 13:56:36.212884 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:36.212866 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" event={"ID":"fcd82cc88301759f1e08e2b014a50651","Type":"ContainerStarted","Data":"cb03afa4c51dff071823c8c1d7f6ff6f927247e4ee5f9d1a4132d0dcb227b90c"} Nov 02 13:56:36.221006 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:36.220991 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" Nov 02 13:56:36.257469 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:36.257447 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 13:56:36.258544 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:36.258531 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" Nov 02 13:56:36.287728 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:36.287705 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 13:56:36.663425 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:36.663401 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:56:36.861745 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:36.861714 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:56:37.093773 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.093704 2577 apiserver.go:52] "Watching apiserver" Nov 02 13:56:37.122294 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.122264 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 13:56:37.138005 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.137978 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 13:56:37.138359 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.138337 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-pstlx","kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal","openshift-multus/network-metrics-daemon-r5vdt","openshift-ovn-kubernetes/ovnkube-node-jg9lw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p","openshift-dns/node-resolver-zgxlv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal","openshift-multus/multus-additional-cni-plugins-ttcsk","openshift-cluster-node-tuning-operator/tuned-v47w5","openshift-image-registry/node-ca-2sxls","openshift-multus/multus-8xfsb","openshift-network-operator/iptables-alerter-sdcn4","kube-system/global-pull-secret-syncer-kv25k","kube-system/konnectivity-agent-5vmgp"] Nov 02 13:56:37.140908 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.140889 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:37.141012 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:37.140990 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv25k" podUID="d1e29491-8fb3-4918-b930-796653f412ec" Nov 02 13:56:37.142921 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.142903 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:37.143031 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:37.142986 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5vdt" podUID="4441e34f-75c1-40d0-9d06-af34fd3fbb50" Nov 02 13:56:37.145041 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.145021 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.147222 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.147205 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.149267 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.149247 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zgxlv" Nov 02 13:56:37.151637 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.151620 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.153458 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.153440 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:56:37.153554 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:37.153497 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pstlx" podUID="7f34d424-89e7-4881-bd4f-8e77566169ed" Nov 02 13:56:37.155647 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.155624 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5vmgp" Nov 02 13:56:37.158496 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.158480 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2sxls" Nov 02 13:56:37.160451 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.160429 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t4wpm\"" Nov 02 13:56:37.160561 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.160471 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.164812 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.164785 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sdcn4" Nov 02 13:56:37.167094 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.167077 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.168981 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.168963 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 13:56:37.169075 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.168989 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 13:56:37.169075 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.169030 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 13:56:37.169075 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.168963 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 13:56:37.169214 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.169091 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 13:56:37.169214 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.169161 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 13:56:37.169711 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.169696 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 13:56:37.169798 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.169751 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tzjpp" Nov 02 13:56:37.222715 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.222694 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 13:56:37.232593 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.232571 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-multus-socket-dir-parent\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.232721 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.232598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-etc-kubernetes\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.232721 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.232612 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:37.232721 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.232653 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvxgk\" (UniqueName: \"kubernetes.io/projected/4441e34f-75c1-40d0-9d06-af34fd3fbb50-kube-api-access-vvxgk\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:37.232721 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.232700 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-host\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.232880 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.232735 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-var-lib-cni-multus\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.232880 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.232758 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/462bec67-536f-47f7-a649-860deadfa79d-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.232880 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.232795 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1e29491-8fb3-4918-b930-796653f412ec-kubelet-config\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:37.232880 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.232820 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-socket-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.232880 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.232863 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-run-netns\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.233073 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.232883 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s78vb\" (UniqueName: \"kubernetes.io/projected/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-kube-api-access-s78vb\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.233073 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.232898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/462bec67-536f-47f7-a649-860deadfa79d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.233073 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.232928 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f651d09a-35a5-4a64-8de3-b824dcfd90ef-host-slash\") pod \"iptables-alerter-sdcn4\" (UID: \"f651d09a-35a5-4a64-8de3-b824dcfd90ef\") " pod="openshift-network-operator/iptables-alerter-sdcn4" Nov 02 13:56:37.233073 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.232963 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-systemd-units\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.233073 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.232986 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-var-lib-openvswitch\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.233073 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-etc-openvswitch\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.233073 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233049 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/462bec67-536f-47f7-a649-860deadfa79d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.233273 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233073 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa6a722d-ff18-4837-8169-f1a642a268bb-agent-certs\") pod \"konnectivity-agent-5vmgp\" (UID: \"aa6a722d-ff18-4837-8169-f1a642a268bb\") " pod="kube-system/konnectivity-agent-5vmgp" Nov 02 13:56:37.233273 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233094 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-systemd\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.233273 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233114 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-sys\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.233273 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233134 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-multus-conf-dir\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.233273 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233189 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr8lv\" (UniqueName: \"kubernetes.io/projected/17f96927-431c-4743-8cda-eb1856feab7a-kube-api-access-mr8lv\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.233273 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233210 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/462bec67-536f-47f7-a649-860deadfa79d-system-cni-dir\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.233273 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233239 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/087ad47b-bcc1-4332-9706-8f0ed148252d-hosts-file\") pod \"node-resolver-zgxlv\" (UID: \"087ad47b-bcc1-4332-9706-8f0ed148252d\") " pod="openshift-dns/node-resolver-zgxlv" Nov 02 13:56:37.233273 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233260 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/087ad47b-bcc1-4332-9706-8f0ed148252d-tmp-dir\") pod \"node-resolver-zgxlv\" (UID: \"087ad47b-bcc1-4332-9706-8f0ed148252d\") " pod="openshift-dns/node-resolver-zgxlv" Nov 02 13:56:37.233508 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233281 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72rpv\" (UniqueName: \"kubernetes.io/projected/087ad47b-bcc1-4332-9706-8f0ed148252d-kube-api-access-72rpv\") pod \"node-resolver-zgxlv\" (UID: \"087ad47b-bcc1-4332-9706-8f0ed148252d\") " pod="openshift-dns/node-resolver-zgxlv" Nov 02 13:56:37.233508 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233304 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-log-socket\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.233508 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233326 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.233508 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233351 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-var-lib-kubelet\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.233508 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233381 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/462bec67-536f-47f7-a649-860deadfa79d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.233508 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233404 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6b9z\" (UniqueName: \"kubernetes.io/projected/e9b81de7-b765-4eac-afd2-86d99495af3d-kube-api-access-g6b9z\") pod \"node-ca-2sxls\" (UID: \"e9b81de7-b765-4eac-afd2-86d99495af3d\") " pod="openshift-image-registry/node-ca-2sxls" Nov 02 13:56:37.233508 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233425 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f651d09a-35a5-4a64-8de3-b824dcfd90ef-iptables-alerter-script\") pod \"iptables-alerter-sdcn4\" (UID: \"f651d09a-35a5-4a64-8de3-b824dcfd90ef\") " pod="openshift-network-operator/iptables-alerter-sdcn4" Nov 02 13:56:37.233508 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233496 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:37.233833 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233516 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-slash\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.233833 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233538 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-run-ovn-kubernetes\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.233833 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233551 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-cni-bin\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.233833 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233565 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-ovnkube-script-lib\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.233833 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233579 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-sysctl-d\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.233833 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233601 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q67r\" (UniqueName: \"kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r\") pod \"network-check-target-pstlx\" (UID: \"7f34d424-89e7-4881-bd4f-8e77566169ed\") " pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:56:37.233833 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233629 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-kubelet\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.233833 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233657 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.233833 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233673 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-tmp\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.233833 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233686 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-system-cni-dir\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.233833 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233702 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-node-log\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.233833 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233727 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-ovnkube-config\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.233833 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233749 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-ovn-node-metrics-cert\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.233833 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233771 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-device-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.233833 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233818 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-lib-modules\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.234478 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233842 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmrvz\" (UniqueName: \"kubernetes.io/projected/f651d09a-35a5-4a64-8de3-b824dcfd90ef-kube-api-access-vmrvz\") pod \"iptables-alerter-sdcn4\" (UID: \"f651d09a-35a5-4a64-8de3-b824dcfd90ef\") " pod="openshift-network-operator/iptables-alerter-sdcn4" Nov 02 13:56:37.234478 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233872 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-cni-netd\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.234478 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233897 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-sysctl-conf\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.234478 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.233911 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-cnibin\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.234478 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234042 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-hostroot\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.234478 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234166 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-run-multus-certs\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.234478 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234201 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9b81de7-b765-4eac-afd2-86d99495af3d-host\") pod \"node-ca-2sxls\" (UID: \"e9b81de7-b765-4eac-afd2-86d99495af3d\") " pod="openshift-image-registry/node-ca-2sxls" Nov 02 13:56:37.234478 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234229 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1e29491-8fb3-4918-b930-796653f412ec-dbus\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:37.234478 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234259 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.234478 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234300 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-run-k8s-cni-cncf-io\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.234478 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234328 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a722d-ff18-4837-8169-f1a642a268bb-konnectivity-ca\") pod \"konnectivity-agent-5vmgp\" (UID: \"aa6a722d-ff18-4837-8169-f1a642a268bb\") " pod="kube-system/konnectivity-agent-5vmgp" Nov 02 13:56:37.234478 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-run-systemd\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.234478 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-run-openvswitch\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.234478 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234400 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvxbp\" (UniqueName: \"kubernetes.io/projected/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-kube-api-access-wvxbp\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.234478 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234414 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17f96927-431c-4743-8cda-eb1856feab7a-cni-binary-copy\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.234478 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234439 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/17f96927-431c-4743-8cda-eb1856feab7a-multus-daemon-config\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.234478 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234454 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-run-ovn\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.235222 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234468 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-run\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.235222 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234481 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-tuned\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.235222 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234505 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-run-netns\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.235222 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234528 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-env-overrides\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.235222 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234549 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-modprobe-d\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.235222 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234569 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-sysconfig\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.235222 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234606 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-kubernetes\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.235222 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234619 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-var-lib-kubelet\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.235222 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234632 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/462bec67-536f-47f7-a649-860deadfa79d-cnibin\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.235222 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234646 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgqq8\" (UniqueName: \"kubernetes.io/projected/f57dae1b-545a-42d4-92f8-19c79e9de22d-kube-api-access-qgqq8\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.235222 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234668 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-multus-cni-dir\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.235222 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234686 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-var-lib-cni-bin\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.235222 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234711 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqtvw\" (UniqueName: \"kubernetes.io/projected/462bec67-536f-47f7-a649-860deadfa79d-kube-api-access-mqtvw\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.235222 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e9b81de7-b765-4eac-afd2-86d99495af3d-serviceca\") pod \"node-ca-2sxls\" (UID: \"e9b81de7-b765-4eac-afd2-86d99495af3d\") " pod="openshift-image-registry/node-ca-2sxls" Nov 02 13:56:37.235222 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234751 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-registration-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.235222 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.234765 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-sys-fs\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.247151 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.247116 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tzjpp" Nov 02 13:56:37.256453 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.256418 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 13:56:37.256572 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.256519 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 13:56:37.256938 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.256921 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 13:56:37.257340 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.257326 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:56:37.257418 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.257394 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 13:56:37.257511 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.257498 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 13:56:37.257657 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.257644 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 13:56:37.257717 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.257707 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 13:56:37.257864 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.257850 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 13:56:37.257923 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.257903 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 13:56:37.257987 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.257926 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 13:56:37.258041 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.258026 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 13:56:37.258102 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.258086 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m5vzv\"" Nov 02 13:56:37.258154 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.258139 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:56:37.258199 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.258170 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 13:56:37.258247 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.258208 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 13:56:37.258293 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.258258 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 13:56:37.258337 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.258303 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 13:56:37.258337 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.258326 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 13:56:37.258418 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.258353 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 13:56:37.258418 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.258383 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 13:56:37.259577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.259470 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-75928\"" Nov 02 13:56:37.259577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.259478 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7mgxc\"" Nov 02 13:56:37.259577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.259514 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 13:56:37.259577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.259523 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-cd6m4\"" Nov 02 13:56:37.259577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.259522 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2zrsm\"" Nov 02 13:56:37.259577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.259514 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4gjwh\"" Nov 02 13:56:37.259577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.259530 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wlrs4\"" Nov 02 13:56:37.259577 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.259573 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kftg9\"" Nov 02 13:56:37.335057 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335030 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/462bec67-536f-47f7-a649-860deadfa79d-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.335206 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335067 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1e29491-8fb3-4918-b930-796653f412ec-kubelet-config\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:37.335206 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335082 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-socket-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.335206 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335164 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1e29491-8fb3-4918-b930-796653f412ec-kubelet-config\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:37.335206 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335192 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-socket-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.335338 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335228 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-run-netns\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.335338 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s78vb\" (UniqueName: \"kubernetes.io/projected/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-kube-api-access-s78vb\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.335338 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335295 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-run-netns\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.335431 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335341 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/462bec67-536f-47f7-a649-860deadfa79d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.335431 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335369 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f651d09a-35a5-4a64-8de3-b824dcfd90ef-host-slash\") pod \"iptables-alerter-sdcn4\" (UID: \"f651d09a-35a5-4a64-8de3-b824dcfd90ef\") " pod="openshift-network-operator/iptables-alerter-sdcn4" Nov 02 13:56:37.335431 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335397 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-systemd-units\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.335431 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335420 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-var-lib-openvswitch\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.335576 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335442 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-etc-openvswitch\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.335576 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335446 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f651d09a-35a5-4a64-8de3-b824dcfd90ef-host-slash\") pod \"iptables-alerter-sdcn4\" (UID: \"f651d09a-35a5-4a64-8de3-b824dcfd90ef\") " pod="openshift-network-operator/iptables-alerter-sdcn4" Nov 02 13:56:37.335576 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335465 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/462bec67-536f-47f7-a649-860deadfa79d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.335576 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335501 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-var-lib-openvswitch\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.335576 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335515 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa6a722d-ff18-4837-8169-f1a642a268bb-agent-certs\") pod \"konnectivity-agent-5vmgp\" (UID: \"aa6a722d-ff18-4837-8169-f1a642a268bb\") " pod="kube-system/konnectivity-agent-5vmgp" Nov 02 13:56:37.335576 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335540 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-systemd\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.335576 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335573 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-etc-openvswitch\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.335576 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335574 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/462bec67-536f-47f7-a649-860deadfa79d-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.335816 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335564 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-sys\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.335816 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-multus-conf-dir\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.335816 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335542 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-systemd-units\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.335816 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335640 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr8lv\" (UniqueName: \"kubernetes.io/projected/17f96927-431c-4743-8cda-eb1856feab7a-kube-api-access-mr8lv\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.335816 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335656 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/462bec67-536f-47f7-a649-860deadfa79d-system-cni-dir\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.335816 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/087ad47b-bcc1-4332-9706-8f0ed148252d-hosts-file\") pod \"node-resolver-zgxlv\" (UID: \"087ad47b-bcc1-4332-9706-8f0ed148252d\") " pod="openshift-dns/node-resolver-zgxlv" Nov 02 13:56:37.335816 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335682 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/087ad47b-bcc1-4332-9706-8f0ed148252d-tmp-dir\") pod \"node-resolver-zgxlv\" (UID: \"087ad47b-bcc1-4332-9706-8f0ed148252d\") " pod="openshift-dns/node-resolver-zgxlv" Nov 02 13:56:37.335816 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72rpv\" (UniqueName: \"kubernetes.io/projected/087ad47b-bcc1-4332-9706-8f0ed148252d-kube-api-access-72rpv\") pod \"node-resolver-zgxlv\" (UID: \"087ad47b-bcc1-4332-9706-8f0ed148252d\") " pod="openshift-dns/node-resolver-zgxlv" Nov 02 13:56:37.335816 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-log-socket\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.335816 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.335816 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-var-lib-kubelet\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.335816 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/462bec67-536f-47f7-a649-860deadfa79d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.335816 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335766 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6b9z\" (UniqueName: \"kubernetes.io/projected/e9b81de7-b765-4eac-afd2-86d99495af3d-kube-api-access-g6b9z\") pod \"node-ca-2sxls\" (UID: \"e9b81de7-b765-4eac-afd2-86d99495af3d\") " pod="openshift-image-registry/node-ca-2sxls" Nov 02 13:56:37.335816 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f651d09a-35a5-4a64-8de3-b824dcfd90ef-iptables-alerter-script\") pod \"iptables-alerter-sdcn4\" (UID: \"f651d09a-35a5-4a64-8de3-b824dcfd90ef\") " pod="openshift-network-operator/iptables-alerter-sdcn4" Nov 02 13:56:37.335816 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335807 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:37.335816 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335618 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-systemd\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.336377 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-slash\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.336377 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335848 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-run-ovn-kubernetes\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.336377 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335857 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/462bec67-536f-47f7-a649-860deadfa79d-system-cni-dir\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.336377 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335869 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-cni-bin\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.336377 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335874 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-sys\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.336377 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335893 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/087ad47b-bcc1-4332-9706-8f0ed148252d-hosts-file\") pod \"node-resolver-zgxlv\" (UID: \"087ad47b-bcc1-4332-9706-8f0ed148252d\") " pod="openshift-dns/node-resolver-zgxlv" Nov 02 13:56:37.336377 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335899 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-ovnkube-script-lib\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.336377 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335923 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-sysctl-d\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.336377 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335957 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q67r\" (UniqueName: \"kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r\") pod \"network-check-target-pstlx\" (UID: \"7f34d424-89e7-4881-bd4f-8e77566169ed\") " pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:56:37.336377 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.335668 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-multus-conf-dir\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.336377 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:37.336358 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:37.336377 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.336372 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-run-ovn-kubernetes\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.336913 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.336382 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.336913 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.336411 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-cni-bin\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.336913 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.336427 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-slash\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.336913 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:37.336459 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret podName:d1e29491-8fb3-4918-b930-796653f412ec nodeName:}" failed. No retries permitted until 2026-11-02 13:56:37.836422061 +0000 UTC m=+3.120538249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret") pod "global-pull-secret-syncer-kv25k" (UID: "d1e29491-8fb3-4918-b930-796653f412ec") : object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:37.336913 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.336465 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-var-lib-kubelet\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.336913 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.336475 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-log-socket\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.336913 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.336555 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-sysctl-d\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.336913 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.336637 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-kubelet\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.336913 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.336673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.336913 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.336697 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-tmp\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.336913 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.336725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-system-cni-dir\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.336913 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.336752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-node-log\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.336913 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.336809 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.336913 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.336856 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-ovnkube-config\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.336913 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.336890 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-node-log\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.336913 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.336900 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-ovn-node-metrics-cert\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.337586 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.337130 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/462bec67-536f-47f7-a649-860deadfa79d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.337586 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.337170 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-device-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.337586 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.337203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-lib-modules\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.337586 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.337238 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmrvz\" (UniqueName: \"kubernetes.io/projected/f651d09a-35a5-4a64-8de3-b824dcfd90ef-kube-api-access-vmrvz\") pod \"iptables-alerter-sdcn4\" (UID: \"f651d09a-35a5-4a64-8de3-b824dcfd90ef\") " pod="openshift-network-operator/iptables-alerter-sdcn4" Nov 02 13:56:37.337586 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.337270 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-cni-netd\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.337586 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.337312 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/462bec67-536f-47f7-a649-860deadfa79d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.337586 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.337310 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 13:56:37.337586 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.337316 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-sysctl-conf\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.337586 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.337413 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-lib-modules\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.337586 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.337524 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-cnibin\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.337586 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.337533 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-sysctl-conf\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.338091 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.337416 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-cnibin\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.338091 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.337602 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-device-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.338091 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.337622 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-system-cni-dir\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.338091 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.337797 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-ovnkube-script-lib\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.338091 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.337865 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-cni-netd\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.338091 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.337919 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-kubelet\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.338351 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338120 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-ovnkube-config\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.338351 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338165 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-hostroot\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.338351 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-run-multus-certs\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.338351 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338229 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9b81de7-b765-4eac-afd2-86d99495af3d-host\") pod \"node-ca-2sxls\" (UID: \"e9b81de7-b765-4eac-afd2-86d99495af3d\") " pod="openshift-image-registry/node-ca-2sxls" Nov 02 13:56:37.338351 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338242 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-hostroot\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.338351 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338254 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1e29491-8fb3-4918-b930-796653f412ec-dbus\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:37.338351 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338295 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-run-multus-certs\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.338351 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338321 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/462bec67-536f-47f7-a649-860deadfa79d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.338351 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338340 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.338720 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-run-k8s-cni-cncf-io\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.338720 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338445 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a722d-ff18-4837-8169-f1a642a268bb-konnectivity-ca\") pod \"konnectivity-agent-5vmgp\" (UID: \"aa6a722d-ff18-4837-8169-f1a642a268bb\") " pod="kube-system/konnectivity-agent-5vmgp" Nov 02 13:56:37.338720 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338502 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-run-systemd\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.338720 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-run-openvswitch\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.338720 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvxbp\" (UniqueName: \"kubernetes.io/projected/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-kube-api-access-wvxbp\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.338720 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338641 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17f96927-431c-4743-8cda-eb1856feab7a-cni-binary-copy\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.338720 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338650 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/087ad47b-bcc1-4332-9706-8f0ed148252d-tmp-dir\") pod \"node-resolver-zgxlv\" (UID: \"087ad47b-bcc1-4332-9706-8f0ed148252d\") " pod="openshift-dns/node-resolver-zgxlv" Nov 02 13:56:37.338720 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/17f96927-431c-4743-8cda-eb1856feab7a-multus-daemon-config\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.338720 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338703 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-run-ovn\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.339159 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338725 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1e29491-8fb3-4918-b930-796653f412ec-dbus\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:37.339159 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338743 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-run\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.339159 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338789 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-run-openvswitch\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.339159 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338804 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-run\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.339159 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-tuned\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.339159 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338856 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-run-netns\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.339159 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338897 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-env-overrides\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.339159 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.338978 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-modprobe-d\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.339159 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339008 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-sysconfig\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.339159 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339051 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-kubernetes\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.339159 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-var-lib-kubelet\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.339159 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339120 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/462bec67-536f-47f7-a649-860deadfa79d-cnibin\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.339651 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgqq8\" (UniqueName: \"kubernetes.io/projected/f57dae1b-545a-42d4-92f8-19c79e9de22d-kube-api-access-qgqq8\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.339651 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339238 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-multus-cni-dir\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.339651 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339290 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-var-lib-cni-bin\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.339651 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339328 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqtvw\" (UniqueName: \"kubernetes.io/projected/462bec67-536f-47f7-a649-860deadfa79d-kube-api-access-mqtvw\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.339651 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339360 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-run-ovn\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.339651 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339370 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e9b81de7-b765-4eac-afd2-86d99495af3d-serviceca\") pod \"node-ca-2sxls\" (UID: \"e9b81de7-b765-4eac-afd2-86d99495af3d\") " pod="openshift-image-registry/node-ca-2sxls" Nov 02 13:56:37.339651 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339403 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-sysconfig\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.339651 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339471 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.339651 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339476 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-registration-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.339651 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339501 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-run-k8s-cni-cncf-io\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.339651 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339534 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-sys-fs\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.339651 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-multus-socket-dir-parent\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.339651 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339583 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/17f96927-431c-4743-8cda-eb1856feab7a-multus-daemon-config\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.339651 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339622 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-etc-kubernetes\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.340278 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339664 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-kubernetes\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.340278 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339680 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-var-lib-cni-bin\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.340278 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339675 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:37.340278 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339707 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-run-netns\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.340278 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339720 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvxgk\" (UniqueName: \"kubernetes.io/projected/4441e34f-75c1-40d0-9d06-af34fd3fbb50-kube-api-access-vvxgk\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:37.340278 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339737 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-run-systemd\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.340278 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-host\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.340278 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339798 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-var-lib-cni-multus\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.340278 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339907 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-var-lib-cni-multus\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.340278 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.340006 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a722d-ff18-4837-8169-f1a642a268bb-konnectivity-ca\") pod \"konnectivity-agent-5vmgp\" (UID: \"aa6a722d-ff18-4837-8169-f1a642a268bb\") " pod="kube-system/konnectivity-agent-5vmgp" Nov 02 13:56:37.340278 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.340129 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17f96927-431c-4743-8cda-eb1856feab7a-cni-binary-copy\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.340278 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.339330 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9b81de7-b765-4eac-afd2-86d99495af3d-host\") pod \"node-ca-2sxls\" (UID: \"e9b81de7-b765-4eac-afd2-86d99495af3d\") " pod="openshift-image-registry/node-ca-2sxls" Nov 02 13:56:37.340278 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.340180 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-var-lib-kubelet\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.340848 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.340348 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-env-overrides\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.340848 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.340358 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-sys-fs\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.340848 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.340422 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-modprobe-d\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.340848 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.340422 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-multus-socket-dir-parent\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.340848 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.340593 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e9b81de7-b765-4eac-afd2-86d99495af3d-serviceca\") pod \"node-ca-2sxls\" (UID: \"e9b81de7-b765-4eac-afd2-86d99495af3d\") " pod="openshift-image-registry/node-ca-2sxls" Nov 02 13:56:37.340848 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.340603 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-etc-kubernetes\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.340848 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.340644 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/462bec67-536f-47f7-a649-860deadfa79d-cnibin\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.340848 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:37.340702 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:37.340848 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.340706 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-registration-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.340848 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.340697 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-multus-cni-dir\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.340848 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.340776 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-host\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.340848 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:37.340787 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs podName:4441e34f-75c1-40d0-9d06-af34fd3fbb50 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:37.840763392 +0000 UTC m=+3.124879575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs") pod "network-metrics-daemon-r5vdt" (UID: "4441e34f-75c1-40d0-9d06-af34fd3fbb50") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:37.342242 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.342217 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f651d09a-35a5-4a64-8de3-b824dcfd90ef-iptables-alerter-script\") pod \"iptables-alerter-sdcn4\" (UID: \"f651d09a-35a5-4a64-8de3-b824dcfd90ef\") " pod="openshift-network-operator/iptables-alerter-sdcn4" Nov 02 13:56:37.342477 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.342455 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-ovn-node-metrics-cert\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.344516 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.344476 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-tmp\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.344516 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.344504 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-tuned\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.344654 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.344636 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa6a722d-ff18-4837-8169-f1a642a268bb-agent-certs\") pod \"konnectivity-agent-5vmgp\" (UID: \"aa6a722d-ff18-4837-8169-f1a642a268bb\") " pod="kube-system/konnectivity-agent-5vmgp" Nov 02 13:56:37.358257 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:37.358236 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 13:56:37.358257 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:37.358257 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 13:56:37.358399 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:37.358267 2577 projected.go:196] Error preparing data for projected volume kube-api-access-7q67r for pod openshift-network-diagnostics/network-check-target-pstlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:37.358399 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:37.358397 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r podName:7f34d424-89e7-4881-bd4f-8e77566169ed nodeName:}" failed. No retries permitted until 2026-11-02 13:56:37.858378161 +0000 UTC m=+3.142494361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7q67r" (UniqueName: "kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r") pod "network-check-target-pstlx" (UID: "7f34d424-89e7-4881-bd4f-8e77566169ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:37.360570 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.360548 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72rpv\" (UniqueName: \"kubernetes.io/projected/087ad47b-bcc1-4332-9706-8f0ed148252d-kube-api-access-72rpv\") pod \"node-resolver-zgxlv\" (UID: \"087ad47b-bcc1-4332-9706-8f0ed148252d\") " pod="openshift-dns/node-resolver-zgxlv" Nov 02 13:56:37.360762 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.360736 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6b9z\" (UniqueName: \"kubernetes.io/projected/e9b81de7-b765-4eac-afd2-86d99495af3d-kube-api-access-g6b9z\") pod \"node-ca-2sxls\" (UID: \"e9b81de7-b765-4eac-afd2-86d99495af3d\") " pod="openshift-image-registry/node-ca-2sxls" Nov 02 13:56:37.360955 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.360920 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr8lv\" (UniqueName: \"kubernetes.io/projected/17f96927-431c-4743-8cda-eb1856feab7a-kube-api-access-mr8lv\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.362330 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.362305 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmrvz\" (UniqueName: \"kubernetes.io/projected/f651d09a-35a5-4a64-8de3-b824dcfd90ef-kube-api-access-vmrvz\") pod \"iptables-alerter-sdcn4\" (UID: \"f651d09a-35a5-4a64-8de3-b824dcfd90ef\") " pod="openshift-network-operator/iptables-alerter-sdcn4" Nov 02 13:56:37.362406 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.362314 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvxbp\" (UniqueName: \"kubernetes.io/projected/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-kube-api-access-wvxbp\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.362574 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.362551 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s78vb\" (UniqueName: \"kubernetes.io/projected/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-kube-api-access-s78vb\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.364041 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.364012 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqtvw\" (UniqueName: \"kubernetes.io/projected/462bec67-536f-47f7-a649-860deadfa79d-kube-api-access-mqtvw\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.364594 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.364574 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgqq8\" (UniqueName: \"kubernetes.io/projected/f57dae1b-545a-42d4-92f8-19c79e9de22d-kube-api-access-qgqq8\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.365392 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.365375 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvxgk\" (UniqueName: \"kubernetes.io/projected/4441e34f-75c1-40d0-9d06-af34fd3fbb50-kube-api-access-vvxgk\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:37.477096 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.477063 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:37.479141 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.479119 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zgxlv" Nov 02 13:56:37.482730 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.482711 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:56:37.482821 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.482807 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:56:37.490856 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.490733 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5vmgp" Nov 02 13:56:37.490856 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:37.490810 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39a3cfd_860d_48e5_bad7_68e4f2b4157a.slice/crio-29fe9d1001a9638fe615caf3d6275e809f9b2ca3de1df95489bc7df1d060bb7a WatchSource:0}: Error finding container 29fe9d1001a9638fe615caf3d6275e809f9b2ca3de1df95489bc7df1d060bb7a: Status 404 returned error can't find the container with id 29fe9d1001a9638fe615caf3d6275e809f9b2ca3de1df95489bc7df1d060bb7a Nov 02 13:56:37.496928 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.496911 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2sxls" Nov 02 13:56:37.499045 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:37.499019 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fb0fb2_293e_43ea_8cae_e2c1235b8d9a.slice/crio-e7d971aaa1499e6cf94b787bff83d1d7f0d6b0bbd3d4c6bb01070a88ff7bf4fd WatchSource:0}: Error finding container e7d971aaa1499e6cf94b787bff83d1d7f0d6b0bbd3d4c6bb01070a88ff7bf4fd: Status 404 returned error can't find the container with id e7d971aaa1499e6cf94b787bff83d1d7f0d6b0bbd3d4c6bb01070a88ff7bf4fd Nov 02 13:56:37.504504 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.504484 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8xfsb" Nov 02 13:56:37.509244 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:37.509221 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57dae1b_545a_42d4_92f8_19c79e9de22d.slice/crio-e46cdba21ca0d6f183fe1812a41b196cc959e006c2c3d19b4265b7cb668e4735 WatchSource:0}: Error finding container e46cdba21ca0d6f183fe1812a41b196cc959e006c2c3d19b4265b7cb668e4735: Status 404 returned error can't find the container with id e46cdba21ca0d6f183fe1812a41b196cc959e006c2c3d19b4265b7cb668e4735 Nov 02 13:56:37.512714 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.512699 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sdcn4" Nov 02 13:56:37.516915 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:37.516890 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6a722d_ff18_4837_8169_f1a642a268bb.slice/crio-bcefcdbc17296b6fd65dc27c30186c616fe97d8ee85e57cc439209990f15be02 WatchSource:0}: Error finding container bcefcdbc17296b6fd65dc27c30186c616fe97d8ee85e57cc439209990f15be02: Status 404 returned error can't find the container with id bcefcdbc17296b6fd65dc27c30186c616fe97d8ee85e57cc439209990f15be02 Nov 02 13:56:37.518994 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:37.518937 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f96927_431c_4743_8cda_eb1856feab7a.slice/crio-1f1464fb8fc7fa5a5bd309e9d751e9661bbbc0323ec5b4aa739bb170e330599f WatchSource:0}: Error finding container 1f1464fb8fc7fa5a5bd309e9d751e9661bbbc0323ec5b4aa739bb170e330599f: Status 404 returned error can't find the container with id 1f1464fb8fc7fa5a5bd309e9d751e9661bbbc0323ec5b4aa739bb170e330599f Nov 02 13:56:37.528990 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.528971 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:56:37.551035 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:56:37.551005 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462bec67_536f_47f7_a649_860deadfa79d.slice/crio-229933466ce509a88d22281825bfeec82aac4a7355957de74b3ca581e336a89b WatchSource:0}: Error finding container 229933466ce509a88d22281825bfeec82aac4a7355957de74b3ca581e336a89b: Status 404 returned error can't find the container with id 229933466ce509a88d22281825bfeec82aac4a7355957de74b3ca581e336a89b Nov 02 13:56:37.844324 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.844286 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:37.844503 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.844332 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:37.844503 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:37.844424 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:37.844503 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:37.844432 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:37.844503 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:37.844470 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret podName:d1e29491-8fb3-4918-b930-796653f412ec nodeName:}" failed. No retries permitted until 2026-11-02 13:56:38.844459493 +0000 UTC m=+4.128575661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret") pod "global-pull-secret-syncer-kv25k" (UID: "d1e29491-8fb3-4918-b930-796653f412ec") : object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:37.844503 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:37.844491 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs podName:4441e34f-75c1-40d0-9d06-af34fd3fbb50 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:38.844476363 +0000 UTC m=+4.128592547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs") pod "network-metrics-daemon-r5vdt" (UID: "4441e34f-75c1-40d0-9d06-af34fd3fbb50") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:37.944801 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:37.944722 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q67r\" (UniqueName: \"kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r\") pod \"network-check-target-pstlx\" (UID: \"7f34d424-89e7-4881-bd4f-8e77566169ed\") " pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:56:37.944957 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:37.944861 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 13:56:37.944957 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:37.944876 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 13:56:37.944957 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:37.944885 2577 projected.go:196] Error preparing data for projected volume kube-api-access-7q67r for pod openshift-network-diagnostics/network-check-target-pstlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:37.945076 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:37.944961 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r podName:7f34d424-89e7-4881-bd4f-8e77566169ed nodeName:}" failed. No retries permitted until 2026-11-02 13:56:38.944928563 +0000 UTC m=+4.229044732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7q67r" (UniqueName: "kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r") pod "network-check-target-pstlx" (UID: "7f34d424-89e7-4881-bd4f-8e77566169ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:38.209768 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:38.209279 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:38.209768 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:38.209405 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5vdt" podUID="4441e34f-75c1-40d0-9d06-af34fd3fbb50" Nov 02 13:56:38.242457 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:38.242386 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttcsk" event={"ID":"462bec67-536f-47f7-a649-860deadfa79d","Type":"ContainerStarted","Data":"229933466ce509a88d22281825bfeec82aac4a7355957de74b3ca581e336a89b"} Nov 02 13:56:38.248722 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:38.248679 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sdcn4" event={"ID":"f651d09a-35a5-4a64-8de3-b824dcfd90ef","Type":"ContainerStarted","Data":"d55253d3ed7a3f63aaf78c2cbc291436e2e99301c6a357735024cfe98b5ebf8f"} Nov 02 13:56:38.250089 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:38.250039 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:51:37 +0000 UTC" deadline="2028-04-21 19:07:00.847412527 +0000 UTC" Nov 02 13:56:38.250089 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:38.250088 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12869h10m22.597327763s" Nov 02 13:56:38.256906 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:38.256878 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2sxls" event={"ID":"e9b81de7-b765-4eac-afd2-86d99495af3d","Type":"ContainerStarted","Data":"c7afc641eccec919d2f64d4055b4c1403a4ce9b4f5fe903aa32bfce6199a87ba"} Nov 02 13:56:38.264149 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:38.264120 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8xfsb" event={"ID":"17f96927-431c-4743-8cda-eb1856feab7a","Type":"ContainerStarted","Data":"1f1464fb8fc7fa5a5bd309e9d751e9661bbbc0323ec5b4aa739bb170e330599f"} Nov 02 13:56:38.266783 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:38.266758 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" event={"ID":"f57dae1b-545a-42d4-92f8-19c79e9de22d","Type":"ContainerStarted","Data":"e46cdba21ca0d6f183fe1812a41b196cc959e006c2c3d19b4265b7cb668e4735"} Nov 02 13:56:38.271284 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:38.271261 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v47w5" event={"ID":"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a","Type":"ContainerStarted","Data":"e7d971aaa1499e6cf94b787bff83d1d7f0d6b0bbd3d4c6bb01070a88ff7bf4fd"} Nov 02 13:56:38.300764 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:38.300732 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" event={"ID":"b39a3cfd-860d-48e5-bad7-68e4f2b4157a","Type":"ContainerStarted","Data":"29fe9d1001a9638fe615caf3d6275e809f9b2ca3de1df95489bc7df1d060bb7a"} Nov 02 13:56:38.326550 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:38.326513 2577 generic.go:356] "Generic (PLEG): container finished" podID="baf43fff2e8eac9bddd0973d4b16dfc0" containerID="74117fd2c594078dbb559421671d1d1d99d5ae5782fc5858452d8ef2e768d56c" exitCode=0 Nov 02 13:56:38.327019 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:38.326781 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" event={"ID":"baf43fff2e8eac9bddd0973d4b16dfc0","Type":"ContainerDied","Data":"74117fd2c594078dbb559421671d1d1d99d5ae5782fc5858452d8ef2e768d56c"} Nov 02 13:56:38.335828 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:38.335801 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" event={"ID":"fcd82cc88301759f1e08e2b014a50651","Type":"ContainerStarted","Data":"939b74f88148ca707b694989b860a78a22deb89e81374a3b076152ba8054f3fd"} Nov 02 13:56:38.374390 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:38.374259 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5vmgp" event={"ID":"aa6a722d-ff18-4837-8169-f1a642a268bb","Type":"ContainerStarted","Data":"bcefcdbc17296b6fd65dc27c30186c616fe97d8ee85e57cc439209990f15be02"} Nov 02 13:56:38.380081 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:38.380036 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zgxlv" event={"ID":"087ad47b-bcc1-4332-9706-8f0ed148252d","Type":"ContainerStarted","Data":"5256243e6aa938799975d72d5774db0b4f1746a1cbea1985cb18b39c80848cfa"} Nov 02 13:56:38.402774 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:38.402718 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" podStartSLOduration=2.402702368 podStartE2EDuration="2.402702368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:38.402265095 +0000 UTC m=+3.686381278" watchObservedRunningTime="2026-11-02 13:56:38.402702368 +0000 UTC m=+3.686818557" Nov 02 13:56:38.868139 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:38.868060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:38.868139 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:38.868125 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:38.868346 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:38.868265 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:38.868346 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:38.868319 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret podName:d1e29491-8fb3-4918-b930-796653f412ec nodeName:}" failed. No retries permitted until 2026-11-02 13:56:40.868304197 +0000 UTC m=+6.152420369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret") pod "global-pull-secret-syncer-kv25k" (UID: "d1e29491-8fb3-4918-b930-796653f412ec") : object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:38.868697 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:38.868662 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:38.868806 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:38.868708 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs podName:4441e34f-75c1-40d0-9d06-af34fd3fbb50 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:40.868694805 +0000 UTC m=+6.152810976 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs") pod "network-metrics-daemon-r5vdt" (UID: "4441e34f-75c1-40d0-9d06-af34fd3fbb50") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:38.969004 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:38.968965 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q67r\" (UniqueName: \"kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r\") pod \"network-check-target-pstlx\" (UID: \"7f34d424-89e7-4881-bd4f-8e77566169ed\") " pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:56:38.969217 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:38.969147 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 13:56:38.969217 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:38.969166 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 13:56:38.969217 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:38.969178 2577 projected.go:196] Error preparing data for projected volume kube-api-access-7q67r for pod openshift-network-diagnostics/network-check-target-pstlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:38.969365 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:38.969232 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r podName:7f34d424-89e7-4881-bd4f-8e77566169ed nodeName:}" failed. No retries permitted until 2026-11-02 13:56:40.969215939 +0000 UTC m=+6.253332122 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7q67r" (UniqueName: "kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r") pod "network-check-target-pstlx" (UID: "7f34d424-89e7-4881-bd4f-8e77566169ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:39.212123 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:39.210068 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:56:39.212123 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:39.210207 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pstlx" podUID="7f34d424-89e7-4881-bd4f-8e77566169ed" Nov 02 13:56:39.212123 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:39.210598 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:39.212123 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:39.210690 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv25k" podUID="d1e29491-8fb3-4918-b930-796653f412ec" Nov 02 13:56:39.250872 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:39.250809 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:51:37 +0000 UTC" deadline="2028-04-02 20:35:26.062705947 +0000 UTC" Nov 02 13:56:39.250872 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:39.250845 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12414h38m46.81186581s" Nov 02 13:56:39.393537 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:39.393499 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" event={"ID":"baf43fff2e8eac9bddd0973d4b16dfc0","Type":"ContainerStarted","Data":"f1db62f7d8d6d2283ffd3ad0cdfbc65062b11c787bc44231f5d0b252149978b2"} Nov 02 13:56:40.209072 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:40.208997 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:40.209240 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:40.209140 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5vdt" podUID="4441e34f-75c1-40d0-9d06-af34fd3fbb50" Nov 02 13:56:40.401176 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:40.400813 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttcsk" event={"ID":"462bec67-536f-47f7-a649-860deadfa79d","Type":"ContainerStarted","Data":"42e327c8ce1d010e2ab541fe24b31f91ac981fe8a1a3a72c160d6a5ba7e8bffc"} Nov 02 13:56:40.473860 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:40.473763 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" podStartSLOduration=5.473743204 podStartE2EDuration="5.473743204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:39.43380652 +0000 UTC m=+4.717922717" watchObservedRunningTime="2026-11-02 13:56:40.473743204 +0000 UTC m=+5.757859397" Nov 02 13:56:40.886628 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:40.886545 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:40.886791 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:40.886637 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:40.886791 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:40.886762 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:40.886889 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:40.886825 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs podName:4441e34f-75c1-40d0-9d06-af34fd3fbb50 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:44.886805441 +0000 UTC m=+10.170921630 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs") pod "network-metrics-daemon-r5vdt" (UID: "4441e34f-75c1-40d0-9d06-af34fd3fbb50") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:40.887594 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:40.887147 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:40.887594 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:40.887211 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret podName:d1e29491-8fb3-4918-b930-796653f412ec nodeName:}" failed. No retries permitted until 2026-11-02 13:56:44.887195055 +0000 UTC m=+10.171311233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret") pod "global-pull-secret-syncer-kv25k" (UID: "d1e29491-8fb3-4918-b930-796653f412ec") : object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:40.987391 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:40.987338 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q67r\" (UniqueName: \"kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r\") pod \"network-check-target-pstlx\" (UID: \"7f34d424-89e7-4881-bd4f-8e77566169ed\") " pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:56:40.987558 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:40.987527 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 13:56:40.987558 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:40.987552 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 13:56:40.987686 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:40.987566 2577 projected.go:196] Error preparing data for projected volume kube-api-access-7q67r for pod openshift-network-diagnostics/network-check-target-pstlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:40.987686 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:40.987631 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r podName:7f34d424-89e7-4881-bd4f-8e77566169ed nodeName:}" failed. No retries permitted until 2026-11-02 13:56:44.987611328 +0000 UTC m=+10.271727523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7q67r" (UniqueName: "kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r") pod "network-check-target-pstlx" (UID: "7f34d424-89e7-4881-bd4f-8e77566169ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:41.210107 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:41.210072 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:41.210273 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:41.210221 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv25k" podUID="d1e29491-8fb3-4918-b930-796653f412ec" Nov 02 13:56:41.210273 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:41.210237 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:56:41.210399 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:41.210375 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pstlx" podUID="7f34d424-89e7-4881-bd4f-8e77566169ed" Nov 02 13:56:41.410762 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:41.410713 2577 generic.go:356] "Generic (PLEG): container finished" podID="462bec67-536f-47f7-a649-860deadfa79d" containerID="42e327c8ce1d010e2ab541fe24b31f91ac981fe8a1a3a72c160d6a5ba7e8bffc" exitCode=0 Nov 02 13:56:41.411168 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:41.410793 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttcsk" event={"ID":"462bec67-536f-47f7-a649-860deadfa79d","Type":"ContainerDied","Data":"42e327c8ce1d010e2ab541fe24b31f91ac981fe8a1a3a72c160d6a5ba7e8bffc"} Nov 02 13:56:41.414108 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:41.414078 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5vmgp" event={"ID":"aa6a722d-ff18-4837-8169-f1a642a268bb","Type":"ContainerStarted","Data":"b706f070ae4ab5317eda2bd4c750df27b4a37f4308c7f75780d253b5d7a3dd24"} Nov 02 13:56:41.454424 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:41.454295 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5vmgp" podStartSLOduration=2.806988091 podStartE2EDuration="6.454275705s" totalImagesPullingTime="3.647287614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:41.453440239 +0000 UTC m=+6.737556432" watchObservedRunningTime="2026-11-02 13:56:41.454275705 +0000 UTC m=+6.738391898" Nov 02 13:56:42.209974 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:42.209427 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:42.209974 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:42.209572 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5vdt" podUID="4441e34f-75c1-40d0-9d06-af34fd3fbb50" Nov 02 13:56:43.212302 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:43.211656 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:43.212302 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:43.211783 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv25k" podUID="d1e29491-8fb3-4918-b930-796653f412ec" Nov 02 13:56:43.212302 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:43.212156 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:56:43.212302 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:43.212246 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pstlx" podUID="7f34d424-89e7-4881-bd4f-8e77566169ed" Nov 02 13:56:43.426986 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:43.426891 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" event={"ID":"f57dae1b-545a-42d4-92f8-19c79e9de22d","Type":"ContainerStarted","Data":"815af2dafdec6826456bfedf3de6870b49f3007f2c2ebf9ee36d9c02afec0848"} Nov 02 13:56:43.428588 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:43.428559 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2sxls" event={"ID":"e9b81de7-b765-4eac-afd2-86d99495af3d","Type":"ContainerStarted","Data":"5020b095aad58b3345ba17acff76547b4ce5729c358ed2242829779f80deadee"} Nov 02 13:56:43.471503 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:43.471443 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2sxls" podStartSLOduration=3.6850980570000003 podStartE2EDuration="8.471425042s" totalImagesPullingTime="4.786326985s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:43.4711622 +0000 UTC m=+8.755278392" watchObservedRunningTime="2026-11-02 13:56:43.471425042 +0000 UTC m=+8.755541236" Nov 02 13:56:43.787726 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:43.786694 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5vmgp" Nov 02 13:56:43.788975 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:43.788729 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5vmgp" Nov 02 13:56:44.209029 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:44.208996 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:44.209213 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:44.209131 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5vdt" podUID="4441e34f-75c1-40d0-9d06-af34fd3fbb50" Nov 02 13:56:44.431660 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:44.431558 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5vmgp" Nov 02 13:56:44.432321 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:44.432022 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5vmgp" Nov 02 13:56:44.571392 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:44.571322 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 13:56:44.917797 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:44.917720 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:44.917933 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:44.917798 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:44.917933 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:44.917878 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:44.918049 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:44.917942 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:44.918049 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:44.917970 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs podName:4441e34f-75c1-40d0-9d06-af34fd3fbb50 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:52.91793236 +0000 UTC m=+18.202048543 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs") pod "network-metrics-daemon-r5vdt" (UID: "4441e34f-75c1-40d0-9d06-af34fd3fbb50") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:44.918049 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:44.918010 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret podName:d1e29491-8fb3-4918-b930-796653f412ec nodeName:}" failed. No retries permitted until 2026-11-02 13:56:52.917995003 +0000 UTC m=+18.202111184 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret") pod "global-pull-secret-syncer-kv25k" (UID: "d1e29491-8fb3-4918-b930-796653f412ec") : object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:45.018748 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:45.018720 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q67r\" (UniqueName: \"kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r\") pod \"network-check-target-pstlx\" (UID: \"7f34d424-89e7-4881-bd4f-8e77566169ed\") " pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:56:45.018907 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:45.018889 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 13:56:45.018985 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:45.018914 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 13:56:45.018985 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:45.018929 2577 projected.go:196] Error preparing data for projected volume kube-api-access-7q67r for pod openshift-network-diagnostics/network-check-target-pstlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:45.019107 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:45.018997 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r podName:7f34d424-89e7-4881-bd4f-8e77566169ed nodeName:}" failed. No retries permitted until 2026-11-02 13:56:53.018979335 +0000 UTC m=+18.303095509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7q67r" (UniqueName: "kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r") pod "network-check-target-pstlx" (UID: "7f34d424-89e7-4881-bd4f-8e77566169ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:45.203822 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:45.203712 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T13:56:44.571345417Z","UUID":"888779f5-cd65-4567-b3fe-d5fe4e242095","Handler":null,"Name":"","Endpoint":""} Nov 02 13:56:45.207974 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:45.205437 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 13:56:45.207974 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:45.205574 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 13:56:45.209776 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:45.209747 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:45.209913 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:45.209864 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv25k" podUID="d1e29491-8fb3-4918-b930-796653f412ec" Nov 02 13:56:45.210011 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:45.209933 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:56:45.210067 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:45.210027 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pstlx" podUID="7f34d424-89e7-4881-bd4f-8e77566169ed" Nov 02 13:56:45.435339 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:45.435052 2577 generic.go:356] "Generic (PLEG): container finished" podID="462bec67-536f-47f7-a649-860deadfa79d" containerID="67eab4a34be1ae5d3e95addfea868eec418679fd4f81fc2f99f110a3f08adf57" exitCode=0 Nov 02 13:56:45.435339 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:45.435267 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttcsk" event={"ID":"462bec67-536f-47f7-a649-860deadfa79d","Type":"ContainerDied","Data":"67eab4a34be1ae5d3e95addfea868eec418679fd4f81fc2f99f110a3f08adf57"} Nov 02 13:56:45.439767 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:45.439281 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8xfsb" event={"ID":"17f96927-431c-4743-8cda-eb1856feab7a","Type":"ContainerStarted","Data":"db939c00621bda289b0879783b9c1dc62841a66905cfa20bc508ae9c391294b6"} Nov 02 13:56:45.442567 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:45.442541 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" event={"ID":"f57dae1b-545a-42d4-92f8-19c79e9de22d","Type":"ContainerStarted","Data":"248fe35cb7f8427d35a6ae6fb55b743a824636d99893900e4d47898b3e2739de"} Nov 02 13:56:45.446681 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:45.446650 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zgxlv" event={"ID":"087ad47b-bcc1-4332-9706-8f0ed148252d","Type":"ContainerStarted","Data":"79189d608dd45ad32f64b3b5ca95b9212c108a70895e0abf58d74d3c5b286838"} Nov 02 13:56:45.530283 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:45.530229 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-8xfsb" podStartSLOduration=3.04772816 podStartE2EDuration="10.53021556s" totalImagesPullingTime="7.4824874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:45.529317947 +0000 UTC m=+10.813434143" watchObservedRunningTime="2026-11-02 13:56:45.53021556 +0000 UTC m=+10.814331754" Nov 02 13:56:45.572507 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:45.572443 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zgxlv" podStartSLOduration=2.987767271 podStartE2EDuration="10.572423984s" totalImagesPullingTime="7.584656713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:45.572017517 +0000 UTC m=+10.856133710" watchObservedRunningTime="2026-11-02 13:56:45.572423984 +0000 UTC m=+10.856540178" Nov 02 13:56:46.209731 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:46.209698 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:46.209885 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:46.209803 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5vdt" podUID="4441e34f-75c1-40d0-9d06-af34fd3fbb50" Nov 02 13:56:46.450785 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:46.450674 2577 generic.go:356] "Generic (PLEG): container finished" podID="462bec67-536f-47f7-a649-860deadfa79d" containerID="f9fde80b5348eff8ac962ca8547e203c3eeec146e4ebe9bf2150b13d0288fafc" exitCode=0 Nov 02 13:56:46.450785 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:46.450766 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttcsk" event={"ID":"462bec67-536f-47f7-a649-860deadfa79d","Type":"ContainerDied","Data":"f9fde80b5348eff8ac962ca8547e203c3eeec146e4ebe9bf2150b13d0288fafc"} Nov 02 13:56:46.453385 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:46.453359 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" event={"ID":"f57dae1b-545a-42d4-92f8-19c79e9de22d","Type":"ContainerStarted","Data":"7ffb054516d585c5f8fc4c6c885fce91e6ed6a20502e9530a45e0a4ad29a9a4c"} Nov 02 13:56:46.545899 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:46.545835 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" podStartSLOduration=3.795673158 podStartE2EDuration="11.545813933s" totalImagesPullingTime="7.750140775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:46.544728439 +0000 UTC m=+11.828844636" watchObservedRunningTime="2026-11-02 13:56:46.545813933 +0000 UTC m=+11.829930160" Nov 02 13:56:47.209921 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:47.209886 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:47.209921 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:47.209905 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:56:47.210128 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:47.210093 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv25k" podUID="d1e29491-8fb3-4918-b930-796653f412ec" Nov 02 13:56:47.210189 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:47.210172 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pstlx" podUID="7f34d424-89e7-4881-bd4f-8e77566169ed" Nov 02 13:56:47.457099 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:47.457064 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v47w5" event={"ID":"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a","Type":"ContainerStarted","Data":"89122763aef2ab0d97a1a67ac35eedc067e46389874ec507edc7776aa1f3e3e4"} Nov 02 13:56:47.462115 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:47.462074 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sdcn4" event={"ID":"f651d09a-35a5-4a64-8de3-b824dcfd90ef","Type":"ContainerStarted","Data":"4cdd500cf120f849788d3eadcf7c25c14b4c0d46acfd1440cda7f3a52705f64c"} Nov 02 13:56:47.505340 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:47.505295 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v47w5" podStartSLOduration=2.969901485 podStartE2EDuration="12.505283788s" totalImagesPullingTime="9.535382303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:47.503825039 +0000 UTC m=+12.787941230" watchObservedRunningTime="2026-11-02 13:56:47.505283788 +0000 UTC m=+12.789399979" Nov 02 13:56:48.209044 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:48.209002 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:48.209209 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:48.209151 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5vdt" podUID="4441e34f-75c1-40d0-9d06-af34fd3fbb50" Nov 02 13:56:48.466969 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:48.466866 2577 generic.go:356] "Generic (PLEG): container finished" podID="462bec67-536f-47f7-a649-860deadfa79d" containerID="995453d0632a5ae3b0f8f2218aa347596528f9c34b62737530b10c651eefa926" exitCode=0 Nov 02 13:56:48.467529 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:48.466969 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttcsk" event={"ID":"462bec67-536f-47f7-a649-860deadfa79d","Type":"ContainerDied","Data":"995453d0632a5ae3b0f8f2218aa347596528f9c34b62737530b10c651eefa926"} Nov 02 13:56:48.542477 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:48.542418 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sdcn4" podStartSLOduration=5.756367914 podStartE2EDuration="13.542399738s" totalImagesPullingTime="7.786031824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:47.550858558 +0000 UTC m=+12.834974752" watchObservedRunningTime="2026-11-02 13:56:48.542399738 +0000 UTC m=+13.826515934" Nov 02 13:56:49.209628 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:49.209591 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:49.209803 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:49.209776 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv25k" podUID="d1e29491-8fb3-4918-b930-796653f412ec" Nov 02 13:56:49.209893 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:49.209880 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:56:49.215006 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:49.210002 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pstlx" podUID="7f34d424-89e7-4881-bd4f-8e77566169ed" Nov 02 13:56:50.209792 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:50.209561 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:50.210171 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:50.209905 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5vdt" podUID="4441e34f-75c1-40d0-9d06-af34fd3fbb50" Nov 02 13:56:50.474681 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:50.474577 2577 generic.go:356] "Generic (PLEG): container finished" podID="462bec67-536f-47f7-a649-860deadfa79d" containerID="01c71ff8650cea18fa1f47e95e0b0a7b3d96eef65d70a5c93da5bd6eabaacf06" exitCode=0 Nov 02 13:56:50.474681 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:50.474622 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttcsk" event={"ID":"462bec67-536f-47f7-a649-860deadfa79d","Type":"ContainerDied","Data":"01c71ff8650cea18fa1f47e95e0b0a7b3d96eef65d70a5c93da5bd6eabaacf06"} Nov 02 13:56:51.209088 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:51.209035 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:51.209271 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:51.209192 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv25k" podUID="d1e29491-8fb3-4918-b930-796653f412ec" Nov 02 13:56:51.209271 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:51.209229 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:56:51.209376 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:51.209308 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pstlx" podUID="7f34d424-89e7-4881-bd4f-8e77566169ed" Nov 02 13:56:51.479449 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:51.479371 2577 generic.go:356] "Generic (PLEG): container finished" podID="462bec67-536f-47f7-a649-860deadfa79d" containerID="15c20953e390f3bc35bfedc10db83555bc196297c5588a19ee0b76a890683e2e" exitCode=0 Nov 02 13:56:51.479449 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:51.479434 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttcsk" event={"ID":"462bec67-536f-47f7-a649-860deadfa79d","Type":"ContainerDied","Data":"15c20953e390f3bc35bfedc10db83555bc196297c5588a19ee0b76a890683e2e"} Nov 02 13:56:52.209942 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:52.209779 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:52.210139 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:52.210117 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5vdt" podUID="4441e34f-75c1-40d0-9d06-af34fd3fbb50" Nov 02 13:56:52.484200 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:52.484164 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" event={"ID":"b39a3cfd-860d-48e5-bad7-68e4f2b4157a","Type":"ContainerStarted","Data":"de030ef7e76244c7892db24961f7b08f06333dd2b54c03e1019b3abf59278fe7"} Nov 02 13:56:52.484643 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:52.484248 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" event={"ID":"b39a3cfd-860d-48e5-bad7-68e4f2b4157a","Type":"ContainerStarted","Data":"0f4139c62a947e1c0d8f054eed5884c7f7acd1ae956034df02cbbc6e8ac85ab7"} Nov 02 13:56:52.484643 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:52.484296 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" event={"ID":"b39a3cfd-860d-48e5-bad7-68e4f2b4157a","Type":"ContainerStarted","Data":"486afd9df19e29e0a993492e01f6407d2d8dadb36662302bea3cdf2314be4d87"} Nov 02 13:56:52.484643 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:52.484346 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" event={"ID":"b39a3cfd-860d-48e5-bad7-68e4f2b4157a","Type":"ContainerStarted","Data":"1d51efe1a8e220e016ecb6ecad55f2df8b946b9d12661d81681d4d3d88c07592"} Nov 02 13:56:52.484643 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:52.484394 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" event={"ID":"b39a3cfd-860d-48e5-bad7-68e4f2b4157a","Type":"ContainerStarted","Data":"84b8b139cda6f3f856fa3a39696b26880dfd27608ae8961e39519148b2c44cab"} Nov 02 13:56:52.484643 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:52.484435 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" event={"ID":"b39a3cfd-860d-48e5-bad7-68e4f2b4157a","Type":"ContainerStarted","Data":"3827853a0f09e83f825141ef5b124d9f9cf1ed4889e3f680c9ffd63f16e2f5e9"} Nov 02 13:56:52.487475 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:52.487453 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttcsk" event={"ID":"462bec67-536f-47f7-a649-860deadfa79d","Type":"ContainerStarted","Data":"a4bc21947f44186309be56a4353270a93068f7cd77fcc4bd489310a3c3a25f16"} Nov 02 13:56:52.977145 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:52.977103 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:52.977315 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:52.977160 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:52.977315 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:52.977265 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:52.977315 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:52.977269 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:52.977440 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:52.977318 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret podName:d1e29491-8fb3-4918-b930-796653f412ec nodeName:}" failed. No retries permitted until 2026-11-02 13:57:08.977303647 +0000 UTC m=+34.261419837 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret") pod "global-pull-secret-syncer-kv25k" (UID: "d1e29491-8fb3-4918-b930-796653f412ec") : object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:52.977440 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:52.977332 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs podName:4441e34f-75c1-40d0-9d06-af34fd3fbb50 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:08.977326126 +0000 UTC m=+34.261442297 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs") pod "network-metrics-daemon-r5vdt" (UID: "4441e34f-75c1-40d0-9d06-af34fd3fbb50") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:53.077565 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:53.077516 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q67r\" (UniqueName: \"kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r\") pod \"network-check-target-pstlx\" (UID: \"7f34d424-89e7-4881-bd4f-8e77566169ed\") " pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:56:53.077736 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:53.077647 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 13:56:53.077736 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:53.077660 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 13:56:53.077736 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:53.077671 2577 projected.go:196] Error preparing data for projected volume kube-api-access-7q67r for pod openshift-network-diagnostics/network-check-target-pstlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:53.077736 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:53.077713 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r podName:7f34d424-89e7-4881-bd4f-8e77566169ed nodeName:}" failed. No retries permitted until 2026-11-02 13:57:09.077701771 +0000 UTC m=+34.361817955 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7q67r" (UniqueName: "kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r") pod "network-check-target-pstlx" (UID: "7f34d424-89e7-4881-bd4f-8e77566169ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:53.209479 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:53.209441 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:56:53.209637 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:53.209573 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pstlx" podUID="7f34d424-89e7-4881-bd4f-8e77566169ed" Nov 02 13:56:53.209637 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:53.209614 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:53.209760 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:53.209698 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv25k" podUID="d1e29491-8fb3-4918-b930-796653f412ec" Nov 02 13:56:54.209072 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:54.209035 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:54.209432 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:54.209146 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5vdt" podUID="4441e34f-75c1-40d0-9d06-af34fd3fbb50" Nov 02 13:56:54.496679 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:54.496602 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" event={"ID":"b39a3cfd-860d-48e5-bad7-68e4f2b4157a","Type":"ContainerStarted","Data":"b67afcada46edce709d676b1fd2450ebe8af8422a77cc723d2a64ac7575a2d65"} Nov 02 13:56:55.210161 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:55.210118 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:55.210607 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:55.210249 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv25k" podUID="d1e29491-8fb3-4918-b930-796653f412ec" Nov 02 13:56:55.210607 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:55.210295 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:56:55.210607 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:55.210383 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pstlx" podUID="7f34d424-89e7-4881-bd4f-8e77566169ed" Nov 02 13:56:56.209116 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:56.209078 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:56.209262 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:56.209187 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5vdt" podUID="4441e34f-75c1-40d0-9d06-af34fd3fbb50" Nov 02 13:56:56.504738 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:56.504655 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" event={"ID":"b39a3cfd-860d-48e5-bad7-68e4f2b4157a","Type":"ContainerStarted","Data":"b81add75de0b3077db20016ad51a4d7a09cf05135466764d7f07df16cd80a980"} Nov 02 13:56:56.505458 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:56.504965 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:56.505458 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:56.505032 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:56.520336 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:56.520314 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:56.603249 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:56.603194 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ttcsk" podStartSLOduration=11.208063605 podStartE2EDuration="21.603181211s" totalImagesPullingTime="10.395117606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:52.546368126 +0000 UTC m=+17.830484319" watchObservedRunningTime="2026-11-02 13:56:56.603181211 +0000 UTC m=+21.887297400" Nov 02 13:56:56.603672 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:56.603650 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" podStartSLOduration=7.20007798 podStartE2EDuration="21.603642329s" totalImagesPullingTime="14.403564349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:56.603155983 +0000 UTC m=+21.887272176" watchObservedRunningTime="2026-11-02 13:56:56.603642329 +0000 UTC m=+21.887758574" Nov 02 13:56:57.209253 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:57.209207 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:56:57.209415 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:57.209336 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pstlx" podUID="7f34d424-89e7-4881-bd4f-8e77566169ed" Nov 02 13:56:57.209415 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:57.209384 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:57.209490 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:57.209473 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv25k" podUID="d1e29491-8fb3-4918-b930-796653f412ec" Nov 02 13:56:57.507701 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:57.507625 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:57.522102 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:57.522079 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:56:58.209169 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:58.209140 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:58.209302 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:58.209256 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5vdt" podUID="4441e34f-75c1-40d0-9d06-af34fd3fbb50" Nov 02 13:56:58.493376 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:58.493303 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kv25k"] Nov 02 13:56:58.493524 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:58.493390 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:56:58.493524 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:58.493465 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv25k" podUID="d1e29491-8fb3-4918-b930-796653f412ec" Nov 02 13:56:58.503429 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:58.503401 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pstlx"] Nov 02 13:56:58.503572 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:58.503493 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:56:58.503635 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:58.503593 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pstlx" podUID="7f34d424-89e7-4881-bd4f-8e77566169ed" Nov 02 13:56:58.503940 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:58.503915 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r5vdt"] Nov 02 13:56:58.509234 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:56:58.509216 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:56:58.509518 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:56:58.509301 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5vdt" podUID="4441e34f-75c1-40d0-9d06-af34fd3fbb50" Nov 02 13:57:00.209872 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:00.209693 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:57:00.210361 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:00.209729 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:57:00.210361 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:00.209939 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pstlx" podUID="7f34d424-89e7-4881-bd4f-8e77566169ed" Nov 02 13:57:00.210361 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:00.209740 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:57:00.210361 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:00.210048 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5vdt" podUID="4441e34f-75c1-40d0-9d06-af34fd3fbb50" Nov 02 13:57:00.210361 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:00.210088 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv25k" podUID="d1e29491-8fb3-4918-b930-796653f412ec" Nov 02 13:57:02.209553 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:02.209522 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:57:02.210097 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:02.209525 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:57:02.210097 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:02.209633 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5vdt" podUID="4441e34f-75c1-40d0-9d06-af34fd3fbb50" Nov 02 13:57:02.210097 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:02.209738 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv25k" podUID="d1e29491-8fb3-4918-b930-796653f412ec" Nov 02 13:57:02.210097 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:02.209525 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:57:02.210097 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:02.209842 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pstlx" podUID="7f34d424-89e7-4881-bd4f-8e77566169ed" Nov 02 13:57:03.562905 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.562877 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeReady" Nov 02 13:57:03.563309 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.563013 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 13:57:03.611589 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.611557 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mbpqq"] Nov 02 13:57:03.614189 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.614168 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mbpqq" Nov 02 13:57:03.615590 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.615562 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-frx4v"] Nov 02 13:57:03.618509 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.618489 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-clfrl"] Nov 02 13:57:03.618631 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.618609 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-frx4v" Nov 02 13:57:03.621087 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.621069 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5"] Nov 02 13:57:03.621186 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.621171 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-clfrl" Nov 02 13:57:03.623674 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.623624 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b64d99df4-mtsvs"] Nov 02 13:57:03.623759 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.623715 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" Nov 02 13:57:03.626551 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.626532 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-579f678cff-m54g4"] Nov 02 13:57:03.626672 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.626657 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.629597 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.629577 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm"] Nov 02 13:57:03.630722 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.630701 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.634254 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.634235 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq"] Nov 02 13:57:03.634352 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.634338 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" Nov 02 13:57:03.637060 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.637045 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5hfrp"] Nov 02 13:57:03.637184 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.637165 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:03.639010 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.638942 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 13:57:03.639010 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.638980 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 13:57:03.639813 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.639799 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fg7tg"] Nov 02 13:57:03.639903 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.639890 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:03.642532 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.642515 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-hvxsl"] Nov 02 13:57:03.642646 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.642630 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:03.645638 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.645621 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7899fcdf65-hl87t"] Nov 02 13:57:03.645776 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.645753 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:03.652300 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.649102 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xrfvm"] Nov 02 13:57:03.652300 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.650626 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:03.652300 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.652107 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 13:57:03.656136 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.656115 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8"] Nov 02 13:57:03.656655 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.656634 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xrfvm" Nov 02 13:57:03.663009 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.659289 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-ljkcn"] Nov 02 13:57:03.663009 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.659360 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:03.663867 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.663846 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cp4vx"] Nov 02 13:57:03.665176 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.664018 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ljkcn" Nov 02 13:57:03.667447 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.667428 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-f9kv5"] Nov 02 13:57:03.667561 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.667545 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:03.670698 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.670661 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mbpqq"] Nov 02 13:57:03.670698 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.670698 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-frx4v"] Nov 02 13:57:03.670877 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.670720 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5"] Nov 02 13:57:03.670877 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.670740 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm"] Nov 02 13:57:03.670877 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.670760 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq"] Nov 02 13:57:03.670877 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.670779 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fg7tg"] Nov 02 13:57:03.670877 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.670800 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5hfrp"] Nov 02 13:57:03.670877 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.670816 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-ljkcn"] Nov 02 13:57:03.671206 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.670889 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f9kv5" Nov 02 13:57:03.677466 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.677444 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f9kv5"] Nov 02 13:57:03.684702 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.684678 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-clfrl"] Nov 02 13:57:03.686023 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.686001 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8"] Nov 02 13:57:03.686709 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.686684 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7899fcdf65-hl87t"] Nov 02 13:57:03.687482 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.687461 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cp4vx"] Nov 02 13:57:03.695885 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.692310 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xrfvm"] Nov 02 13:57:03.695885 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.692358 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-hvxsl"] Nov 02 13:57:03.697662 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.697636 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b64d99df4-mtsvs"] Nov 02 13:57:03.698439 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.698404 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-579f678cff-m54g4"] Nov 02 13:57:03.744970 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.744934 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 13:57:03.745161 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.745144 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 13:57:03.745204 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.745158 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 13:57:03.745251 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.745205 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jk68j\"" Nov 02 13:57:03.745471 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.745458 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:57:03.745517 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.745495 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 13:57:03.745568 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.745551 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 13:57:03.745568 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.745554 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 13:57:03.747226 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.746192 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 13:57:03.747226 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.746357 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 13:57:03.747226 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.746905 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mbsf9\"" Nov 02 13:57:03.747226 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.747040 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 13:57:03.747226 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.747131 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 13:57:03.747226 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.747045 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 13:57:03.747566 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.747354 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 13:57:03.747566 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.747444 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nkhq6\"" Nov 02 13:57:03.756539 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.756514 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f212522d-ef61-4b05-aee8-538433003f78-ca-trust-extracted\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.756642 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.756542 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:03.756642 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.756560 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:03.756642 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.756578 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd3bd8d9-b081-48bd-be29-582afe6f82fd-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm\" (UID: \"bd3bd8d9-b081-48bd-be29-582afe6f82fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" Nov 02 13:57:03.756791 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.756641 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-bound-sa-token\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.756791 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.756729 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-image-registry-private-configuration\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.756791 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.756756 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-hvxsl\" (UID: \"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:03.756791 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.756772 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b97f6b80-f53b-4fff-97f6-dff42fb5875f-trusted-ca\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:03.756791 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.756788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lvl4\" (UniqueName: \"kubernetes.io/projected/4f856415-9f37-415c-8dd5-65fc1ef08abc-kube-api-access-7lvl4\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:03.756967 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.756813 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-config-volume\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:03.756967 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.756877 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75khg\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-kube-api-access-75khg\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.756967 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.756908 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f856415-9f37-415c-8dd5-65fc1ef08abc-serving-cert\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:03.756967 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.756930 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f856415-9f37-415c-8dd5-65fc1ef08abc-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:03.757080 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.756984 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.757080 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757021 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-installation-pull-secrets\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.757080 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757047 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-bound-sa-token\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.757080 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757070 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fr7m\" (UniqueName: \"kubernetes.io/projected/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-kube-api-access-9fr7m\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:03.757192 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757127 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hbn9\" (UniqueName: \"kubernetes.io/projected/d445fa88-f52d-460f-b89a-61de9cc8d541-kube-api-access-9hbn9\") pod \"kas-connection-checker-685895c7c9-mbpqq\" (UID: \"d445fa88-f52d-460f-b89a-61de9cc8d541\") " pod="kube-system/kas-connection-checker-685895c7c9-mbpqq" Nov 02 13:57:03.757192 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757149 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bwcd\" (UniqueName: \"kubernetes.io/projected/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-kube-api-access-8bwcd\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:03.757192 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757176 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-default-certificate\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:03.757302 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757193 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:03.757302 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757207 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f856415-9f37-415c-8dd5-65fc1ef08abc-tmp\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:03.757302 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757229 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:03.757302 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757243 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-trusted-ca\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.757302 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757263 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-installation-pull-secrets\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.757509 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc0287f6-d102-43a4-80cd-62b3acfb3991-config\") pod \"service-ca-operator-544d749c8c-zv6s5\" (UID: \"fc0287f6-d102-43a4-80cd-62b3acfb3991\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" Nov 02 13:57:03.757509 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757381 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4zbv\" (UniqueName: \"kubernetes.io/projected/7157562b-09b1-44a1-95f8-f29c7e62e475-kube-api-access-z4zbv\") pod \"kas-connection-checker-685895c7c9-clfrl\" (UID: \"7157562b-09b1-44a1-95f8-f29c7e62e475\") " pod="kube-system/kas-connection-checker-685895c7c9-clfrl" Nov 02 13:57:03.757509 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757408 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebd0d655-5116-4a35-89ff-f7575026d4ea-ca-trust-extracted\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.757509 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757428 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-trusted-ca\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.757509 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757466 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xctln\" (UniqueName: \"kubernetes.io/projected/fc0287f6-d102-43a4-80cd-62b3acfb3991-kube-api-access-xctln\") pod \"service-ca-operator-544d749c8c-zv6s5\" (UID: \"fc0287f6-d102-43a4-80cd-62b3acfb3991\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" Nov 02 13:57:03.757509 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757498 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7ms9\" (UniqueName: \"kubernetes.io/projected/bd3bd8d9-b081-48bd-be29-582afe6f82fd-kube-api-access-d7ms9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm\" (UID: \"bd3bd8d9-b081-48bd-be29-582afe6f82fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" Nov 02 13:57:03.757682 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757523 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc0287f6-d102-43a4-80cd-62b3acfb3991-serving-cert\") pod \"service-ca-operator-544d749c8c-zv6s5\" (UID: \"fc0287f6-d102-43a4-80cd-62b3acfb3991\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" Nov 02 13:57:03.757682 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757567 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bd3bd8d9-b081-48bd-be29-582afe6f82fd-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm\" (UID: \"bd3bd8d9-b081-48bd-be29-582afe6f82fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" Nov 02 13:57:03.757682 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757602 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:03.757682 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757621 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f856415-9f37-415c-8dd5-65fc1ef08abc-service-ca-bundle\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:03.757682 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757662 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b97f6b80-f53b-4fff-97f6-dff42fb5875f-serving-cert\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:03.757682 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757679 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.757903 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757694 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:03.757903 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757709 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5p6j\" (UniqueName: \"kubernetes.io/projected/b97f6b80-f53b-4fff-97f6-dff42fb5875f-kube-api-access-t5p6j\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:03.757903 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757740 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p67v9\" (UniqueName: \"kubernetes.io/projected/7120b45c-1c5b-4c7c-863f-1902f96c00f5-kube-api-access-p67v9\") pod \"ingress-canary-f9kv5\" (UID: \"7120b45c-1c5b-4c7c-863f-1902f96c00f5\") " pod="openshift-ingress-canary/ingress-canary-f9kv5" Nov 02 13:57:03.757903 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757818 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9prcq\" (UniqueName: \"kubernetes.io/projected/a8133c4b-08bc-443b-aa8c-d914da38a2a5-kube-api-access-9prcq\") pod \"network-check-source-6f78d59f99-ljkcn\" (UID: \"a8133c4b-08bc-443b-aa8c-d914da38a2a5\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ljkcn" Nov 02 13:57:03.757903 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw75f\" (UniqueName: \"kubernetes.io/projected/87b5876f-0012-4bdf-b037-25070684476c-kube-api-access-bw75f\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:03.757903 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757867 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-image-registry-private-configuration\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.758173 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757929 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b97f6b80-f53b-4fff-97f6-dff42fb5875f-config\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:03.758173 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.757975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghtzt\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-kube-api-access-ghtzt\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.758173 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758007 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-certificates\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.758173 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-registry-certificates\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.758173 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758058 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tss59\" (UniqueName: \"kubernetes.io/projected/cf94ad7e-96a0-4d35-9cb3-ad83f8234281-kube-api-access-tss59\") pod \"volume-data-source-validator-58c9f5f46d-xrfvm\" (UID: \"cf94ad7e-96a0-4d35-9cb3-ad83f8234281\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xrfvm" Nov 02 13:57:03.758173 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758088 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hvxsl\" (UID: \"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:03.758173 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758141 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4f856415-9f37-415c-8dd5-65fc1ef08abc-snapshots\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:03.758173 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758169 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trjsq\" (UniqueName: \"kubernetes.io/projected/a7095b0b-813f-4b60-989f-8a01e2743bbe-kube-api-access-trjsq\") pod \"kas-connection-checker-685895c7c9-frx4v\" (UID: \"a7095b0b-813f-4b60-989f-8a01e2743bbe\") " pod="kube-system/kas-connection-checker-685895c7c9-frx4v" Nov 02 13:57:03.758495 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758204 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7120b45c-1c5b-4c7c-863f-1902f96c00f5-cert\") pod \"ingress-canary-f9kv5\" (UID: \"7120b45c-1c5b-4c7c-863f-1902f96c00f5\") " pod="openshift-ingress-canary/ingress-canary-f9kv5" Nov 02 13:57:03.758495 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758224 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-stats-auth\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:03.758495 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758242 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-tmp-dir\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:03.758495 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758264 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tht9k\" (UniqueName: \"kubernetes.io/projected/b1259f5f-470e-4ca5-8760-37bff1735835-kube-api-access-tht9k\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:03.758495 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758306 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b1259f5f-470e-4ca5-8760-37bff1735835-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:03.758495 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758381 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 13:57:03.758495 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758387 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 13:57:03.758495 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758407 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 13:57:03.758495 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758491 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-qxxgw\"" Nov 02 13:57:03.758804 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758601 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:57:03.758804 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758604 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 13:57:03.758804 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758645 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 13:57:03.758804 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758678 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 13:57:03.758804 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758678 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-w7snh\"" Nov 02 13:57:03.758934 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.758889 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 13:57:03.759196 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759183 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 13:57:03.759246 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759202 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 13:57:03.759246 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759230 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 13:57:03.759338 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759249 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 13:57:03.759338 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759294 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4pvjb\"" Nov 02 13:57:03.759338 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759297 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:57:03.759338 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759329 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 13:57:03.759456 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759388 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 13:57:03.759499 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759487 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 13:57:03.759683 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759672 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 13:57:03.759683 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759680 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 13:57:03.759736 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759706 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 13:57:03.759765 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759734 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:57:03.759797 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759782 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 13:57:03.759797 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759785 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 13:57:03.759860 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759838 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-f9rnl\"" Nov 02 13:57:03.759902 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759865 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 13:57:03.759902 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759885 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:57:03.759902 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759889 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 13:57:03.759902 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759893 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 13:57:03.760072 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759909 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 13:57:03.760072 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.759982 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 13:57:03.760072 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.760005 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 13:57:03.760072 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.760015 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mfgzk\"" Nov 02 13:57:03.760197 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.760128 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bhk6z\"" Nov 02 13:57:03.772992 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.772976 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-lpst2\"" Nov 02 13:57:03.773071 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.773005 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 13:57:03.773071 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.773009 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b9rw8\"" Nov 02 13:57:03.773071 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.773025 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9cgsx\"" Nov 02 13:57:03.773071 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.773010 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7l5rk\"" Nov 02 13:57:03.774001 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.773981 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 13:57:03.774068 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.774055 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 13:57:03.774880 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.774863 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 13:57:03.859142 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859110 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc0287f6-d102-43a4-80cd-62b3acfb3991-config\") pod \"service-ca-operator-544d749c8c-zv6s5\" (UID: \"fc0287f6-d102-43a4-80cd-62b3acfb3991\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" Nov 02 13:57:03.859142 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4zbv\" (UniqueName: \"kubernetes.io/projected/7157562b-09b1-44a1-95f8-f29c7e62e475-kube-api-access-z4zbv\") pod \"kas-connection-checker-685895c7c9-clfrl\" (UID: \"7157562b-09b1-44a1-95f8-f29c7e62e475\") " pod="kube-system/kas-connection-checker-685895c7c9-clfrl" Nov 02 13:57:03.859353 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859160 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebd0d655-5116-4a35-89ff-f7575026d4ea-ca-trust-extracted\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.859353 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859181 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-trusted-ca\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.859353 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859223 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xctln\" (UniqueName: \"kubernetes.io/projected/fc0287f6-d102-43a4-80cd-62b3acfb3991-kube-api-access-xctln\") pod \"service-ca-operator-544d749c8c-zv6s5\" (UID: \"fc0287f6-d102-43a4-80cd-62b3acfb3991\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" Nov 02 13:57:03.859353 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859258 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7ms9\" (UniqueName: \"kubernetes.io/projected/bd3bd8d9-b081-48bd-be29-582afe6f82fd-kube-api-access-d7ms9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm\" (UID: \"bd3bd8d9-b081-48bd-be29-582afe6f82fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" Nov 02 13:57:03.859353 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859294 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc0287f6-d102-43a4-80cd-62b3acfb3991-serving-cert\") pod \"service-ca-operator-544d749c8c-zv6s5\" (UID: \"fc0287f6-d102-43a4-80cd-62b3acfb3991\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" Nov 02 13:57:03.859353 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bd3bd8d9-b081-48bd-be29-582afe6f82fd-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm\" (UID: \"bd3bd8d9-b081-48bd-be29-582afe6f82fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" Nov 02 13:57:03.859353 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859352 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:03.859693 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859380 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f856415-9f37-415c-8dd5-65fc1ef08abc-service-ca-bundle\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:03.859693 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859409 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b97f6b80-f53b-4fff-97f6-dff42fb5875f-serving-cert\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:03.859693 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859433 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.859693 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859459 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:03.859693 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859486 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5p6j\" (UniqueName: \"kubernetes.io/projected/b97f6b80-f53b-4fff-97f6-dff42fb5875f-kube-api-access-t5p6j\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:03.859693 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859511 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67v9\" (UniqueName: \"kubernetes.io/projected/7120b45c-1c5b-4c7c-863f-1902f96c00f5-kube-api-access-p67v9\") pod \"ingress-canary-f9kv5\" (UID: \"7120b45c-1c5b-4c7c-863f-1902f96c00f5\") " pod="openshift-ingress-canary/ingress-canary-f9kv5" Nov 02 13:57:03.859693 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859548 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prcq\" (UniqueName: \"kubernetes.io/projected/a8133c4b-08bc-443b-aa8c-d914da38a2a5-kube-api-access-9prcq\") pod \"network-check-source-6f78d59f99-ljkcn\" (UID: \"a8133c4b-08bc-443b-aa8c-d914da38a2a5\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ljkcn" Nov 02 13:57:03.859693 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw75f\" (UniqueName: \"kubernetes.io/projected/87b5876f-0012-4bdf-b037-25070684476c-kube-api-access-bw75f\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:03.859693 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859607 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-image-registry-private-configuration\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.859693 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859630 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebd0d655-5116-4a35-89ff-f7575026d4ea-ca-trust-extracted\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.859693 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859638 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b97f6b80-f53b-4fff-97f6-dff42fb5875f-config\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:03.859693 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859661 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghtzt\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-kube-api-access-ghtzt\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.859693 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859685 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-certificates\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.860405 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859709 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-registry-certificates\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.860405 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:03.859726 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 13:57:03.860405 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:03.859781 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls podName:1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d nodeName:}" failed. No retries permitted until 2026-11-02 13:57:04.359762039 +0000 UTC m=+29.643878226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4sfb8" (UID: "1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d") : secret "samples-operator-tls" not found Nov 02 13:57:03.860405 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859907 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc0287f6-d102-43a4-80cd-62b3acfb3991-config\") pod \"service-ca-operator-544d749c8c-zv6s5\" (UID: \"fc0287f6-d102-43a4-80cd-62b3acfb3991\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" Nov 02 13:57:03.860405 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860060 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f856415-9f37-415c-8dd5-65fc1ef08abc-service-ca-bundle\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:03.860405 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860094 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bd3bd8d9-b081-48bd-be29-582afe6f82fd-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm\" (UID: \"bd3bd8d9-b081-48bd-be29-582afe6f82fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" Nov 02 13:57:03.860405 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.859737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tss59\" (UniqueName: \"kubernetes.io/projected/cf94ad7e-96a0-4d35-9cb3-ad83f8234281-kube-api-access-tss59\") pod \"volume-data-source-validator-58c9f5f46d-xrfvm\" (UID: \"cf94ad7e-96a0-4d35-9cb3-ad83f8234281\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xrfvm" Nov 02 13:57:03.860405 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hvxsl\" (UID: \"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:03.860405 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860342 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4f856415-9f37-415c-8dd5-65fc1ef08abc-snapshots\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:03.860405 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860373 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trjsq\" (UniqueName: \"kubernetes.io/projected/a7095b0b-813f-4b60-989f-8a01e2743bbe-kube-api-access-trjsq\") pod \"kas-connection-checker-685895c7c9-frx4v\" (UID: \"a7095b0b-813f-4b60-989f-8a01e2743bbe\") " pod="kube-system/kas-connection-checker-685895c7c9-frx4v" Nov 02 13:57:03.860405 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860385 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:03.860405 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860397 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7120b45c-1c5b-4c7c-863f-1902f96c00f5-cert\") pod \"ingress-canary-f9kv5\" (UID: \"7120b45c-1c5b-4c7c-863f-1902f96c00f5\") " pod="openshift-ingress-canary/ingress-canary-f9kv5" Nov 02 13:57:03.861030 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860430 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-stats-auth\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:03.861030 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860455 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-tmp-dir\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:03.861030 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tht9k\" (UniqueName: \"kubernetes.io/projected/b1259f5f-470e-4ca5-8760-37bff1735835-kube-api-access-tht9k\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:03.861030 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:03.860497 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 13:57:03.861030 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:03.860510 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b64d99df4-mtsvs: secret "image-registry-tls" not found Nov 02 13:57:03.861030 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860510 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b1259f5f-470e-4ca5-8760-37bff1735835-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:03.861030 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860538 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f212522d-ef61-4b05-aee8-538433003f78-ca-trust-extracted\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.861030 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:03.860553 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls podName:ebd0d655-5116-4a35-89ff-f7575026d4ea nodeName:}" failed. No retries permitted until 2026-11-02 13:57:04.360538177 +0000 UTC m=+29.644654371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls") pod "image-registry-b64d99df4-mtsvs" (UID: "ebd0d655-5116-4a35-89ff-f7575026d4ea") : secret "image-registry-tls" not found Nov 02 13:57:03.861030 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860579 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:03.861030 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860611 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:03.861030 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860635 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd3bd8d9-b081-48bd-be29-582afe6f82fd-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm\" (UID: \"bd3bd8d9-b081-48bd-be29-582afe6f82fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" Nov 02 13:57:03.861030 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860658 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-bound-sa-token\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.861030 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860686 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-image-registry-private-configuration\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.861030 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860714 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-hvxsl\" (UID: \"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:03.861030 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b97f6b80-f53b-4fff-97f6-dff42fb5875f-trusted-ca\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:03.861030 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860756 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-trusted-ca\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.863155 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lvl4\" (UniqueName: \"kubernetes.io/projected/4f856415-9f37-415c-8dd5-65fc1ef08abc-kube-api-access-7lvl4\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:03.863155 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860808 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-config-volume\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:03.863155 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860840 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f212522d-ef61-4b05-aee8-538433003f78-ca-trust-extracted\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.863155 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75khg\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-kube-api-access-75khg\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.863155 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860856 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-registry-certificates\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.863155 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860880 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f856415-9f37-415c-8dd5-65fc1ef08abc-serving-cert\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:03.863155 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860912 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b97f6b80-f53b-4fff-97f6-dff42fb5875f-config\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:03.863155 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f856415-9f37-415c-8dd5-65fc1ef08abc-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:03.863155 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.860984 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.863155 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.861012 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-installation-pull-secrets\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.863155 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.861035 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-bound-sa-token\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.863155 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:03.861053 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 13:57:03.863155 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.861059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fr7m\" (UniqueName: \"kubernetes.io/projected/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-kube-api-access-9fr7m\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:03.863155 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.861102 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hbn9\" (UniqueName: \"kubernetes.io/projected/d445fa88-f52d-460f-b89a-61de9cc8d541-kube-api-access-9hbn9\") pod \"kas-connection-checker-685895c7c9-mbpqq\" (UID: \"d445fa88-f52d-460f-b89a-61de9cc8d541\") " pod="kube-system/kas-connection-checker-685895c7c9-mbpqq" Nov 02 13:57:03.863155 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.861125 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bwcd\" (UniqueName: \"kubernetes.io/projected/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-kube-api-access-8bwcd\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:03.863155 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:03.861156 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 13:57:03.863155 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:03.861194 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs podName:87b5876f-0012-4bdf-b037-25070684476c nodeName:}" failed. No retries permitted until 2026-11-02 13:57:04.361179715 +0000 UTC m=+29.645295909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs") pod "router-default-7899fcdf65-hl87t" (UID: "87b5876f-0012-4bdf-b037-25070684476c") : secret "router-metrics-certs-default" not found Nov 02 13:57:03.866778 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.861345 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-certificates\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.866778 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:03.861412 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 13:57:03.866778 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.861415 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4f856415-9f37-415c-8dd5-65fc1ef08abc-snapshots\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:03.866778 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:03.861459 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert podName:cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:04.361445034 +0000 UTC m=+29.645561218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hvxsl" (UID: "cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1") : secret "networking-console-plugin-cert" not found Nov 02 13:57:03.866778 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.861632 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f856415-9f37-415c-8dd5-65fc1ef08abc-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:03.866778 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:03.861766 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls podName:ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:04.36175018 +0000 UTC m=+29.645866351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls") pod "dns-default-cp4vx" (UID: "ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e") : secret "dns-default-metrics-tls" not found Nov 02 13:57:03.866778 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:03.861903 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 13:57:03.866778 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:03.861919 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579f678cff-m54g4: secret "image-registry-tls" not found Nov 02 13:57:03.866778 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:03.861976 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls podName:f212522d-ef61-4b05-aee8-538433003f78 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:04.361962938 +0000 UTC m=+29.646079130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls") pod "image-registry-579f678cff-m54g4" (UID: "f212522d-ef61-4b05-aee8-538433003f78") : secret "image-registry-tls" not found Nov 02 13:57:03.866778 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.861156 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-default-certificate\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:03.866778 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.863388 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:03.866778 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.863420 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f856415-9f37-415c-8dd5-65fc1ef08abc-tmp\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:03.866778 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.863471 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:03.866778 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.863504 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-trusted-ca\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.866778 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.863519 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-tmp-dir\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:03.866778 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.863530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-installation-pull-secrets\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.867550 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:03.863695 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle podName:87b5876f-0012-4bdf-b037-25070684476c nodeName:}" failed. No retries permitted until 2026-11-02 13:57:04.363680085 +0000 UTC m=+29.647796276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle") pod "router-default-7899fcdf65-hl87t" (UID: "87b5876f-0012-4bdf-b037-25070684476c") : configmap references non-existent config key: service-ca.crt Nov 02 13:57:03.867550 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.864022 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f856415-9f37-415c-8dd5-65fc1ef08abc-tmp\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:03.867550 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:03.864114 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 13:57:03.867550 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:03.864161 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls podName:b1259f5f-470e-4ca5-8760-37bff1735835 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:04.36414422 +0000 UTC m=+29.648260404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4tgbq" (UID: "b1259f5f-470e-4ca5-8760-37bff1735835") : secret "cluster-monitoring-operator-tls" not found Nov 02 13:57:03.867550 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.864231 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-config-volume\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:03.867550 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.864595 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b97f6b80-f53b-4fff-97f6-dff42fb5875f-trusted-ca\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:03.867550 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.864775 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b1259f5f-470e-4ca5-8760-37bff1735835-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:03.867550 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.864830 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-hvxsl\" (UID: \"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:03.867550 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.865169 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-trusted-ca\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.867550 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.865249 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-image-registry-private-configuration\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.867550 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.865286 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b97f6b80-f53b-4fff-97f6-dff42fb5875f-serving-cert\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:03.867550 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.865553 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-stats-auth\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:03.867550 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.865994 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7120b45c-1c5b-4c7c-863f-1902f96c00f5-cert\") pod \"ingress-canary-f9kv5\" (UID: \"7120b45c-1c5b-4c7c-863f-1902f96c00f5\") " pod="openshift-ingress-canary/ingress-canary-f9kv5" Nov 02 13:57:03.867550 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.866028 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-default-certificate\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:03.867550 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.866456 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd3bd8d9-b081-48bd-be29-582afe6f82fd-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm\" (UID: \"bd3bd8d9-b081-48bd-be29-582afe6f82fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" Nov 02 13:57:03.868248 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.867163 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-installation-pull-secrets\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.868248 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.867238 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f856415-9f37-415c-8dd5-65fc1ef08abc-serving-cert\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:03.868248 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.867266 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc0287f6-d102-43a4-80cd-62b3acfb3991-serving-cert\") pod \"service-ca-operator-544d749c8c-zv6s5\" (UID: \"fc0287f6-d102-43a4-80cd-62b3acfb3991\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" Nov 02 13:57:03.868248 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.867470 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-image-registry-private-configuration\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.868248 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.867855 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-installation-pull-secrets\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.904138 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.904079 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-bound-sa-token\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.904302 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.904283 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5p6j\" (UniqueName: \"kubernetes.io/projected/b97f6b80-f53b-4fff-97f6-dff42fb5875f-kube-api-access-t5p6j\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:03.910628 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.910596 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9prcq\" (UniqueName: \"kubernetes.io/projected/a8133c4b-08bc-443b-aa8c-d914da38a2a5-kube-api-access-9prcq\") pod \"network-check-source-6f78d59f99-ljkcn\" (UID: \"a8133c4b-08bc-443b-aa8c-d914da38a2a5\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ljkcn" Nov 02 13:57:03.911218 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.911194 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lvl4\" (UniqueName: \"kubernetes.io/projected/4f856415-9f37-415c-8dd5-65fc1ef08abc-kube-api-access-7lvl4\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:03.911851 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.911722 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trjsq\" (UniqueName: \"kubernetes.io/projected/a7095b0b-813f-4b60-989f-8a01e2743bbe-kube-api-access-trjsq\") pod \"kas-connection-checker-685895c7c9-frx4v\" (UID: \"a7095b0b-813f-4b60-989f-8a01e2743bbe\") " pod="kube-system/kas-connection-checker-685895c7c9-frx4v" Nov 02 13:57:03.911959 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.911846 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghtzt\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-kube-api-access-ghtzt\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:03.912143 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.912109 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75khg\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-kube-api-access-75khg\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.912329 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.912288 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tht9k\" (UniqueName: \"kubernetes.io/projected/b1259f5f-470e-4ca5-8760-37bff1735835-kube-api-access-tht9k\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:03.912531 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.912506 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xctln\" (UniqueName: \"kubernetes.io/projected/fc0287f6-d102-43a4-80cd-62b3acfb3991-kube-api-access-xctln\") pod \"service-ca-operator-544d749c8c-zv6s5\" (UID: \"fc0287f6-d102-43a4-80cd-62b3acfb3991\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" Nov 02 13:57:03.913085 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.912993 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tss59\" (UniqueName: \"kubernetes.io/projected/cf94ad7e-96a0-4d35-9cb3-ad83f8234281-kube-api-access-tss59\") pod \"volume-data-source-validator-58c9f5f46d-xrfvm\" (UID: \"cf94ad7e-96a0-4d35-9cb3-ad83f8234281\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xrfvm" Nov 02 13:57:03.913393 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.913342 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4zbv\" (UniqueName: \"kubernetes.io/projected/7157562b-09b1-44a1-95f8-f29c7e62e475-kube-api-access-z4zbv\") pod \"kas-connection-checker-685895c7c9-clfrl\" (UID: \"7157562b-09b1-44a1-95f8-f29c7e62e475\") " pod="kube-system/kas-connection-checker-685895c7c9-clfrl" Nov 02 13:57:03.914843 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.914784 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-bound-sa-token\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:03.914843 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.914790 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hbn9\" (UniqueName: \"kubernetes.io/projected/d445fa88-f52d-460f-b89a-61de9cc8d541-kube-api-access-9hbn9\") pod \"kas-connection-checker-685895c7c9-mbpqq\" (UID: \"d445fa88-f52d-460f-b89a-61de9cc8d541\") " pod="kube-system/kas-connection-checker-685895c7c9-mbpqq" Nov 02 13:57:03.915031 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.914927 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw75f\" (UniqueName: \"kubernetes.io/projected/87b5876f-0012-4bdf-b037-25070684476c-kube-api-access-bw75f\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:03.915031 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.914982 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p67v9\" (UniqueName: \"kubernetes.io/projected/7120b45c-1c5b-4c7c-863f-1902f96c00f5-kube-api-access-p67v9\") pod \"ingress-canary-f9kv5\" (UID: \"7120b45c-1c5b-4c7c-863f-1902f96c00f5\") " pod="openshift-ingress-canary/ingress-canary-f9kv5" Nov 02 13:57:03.915133 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.915028 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7ms9\" (UniqueName: \"kubernetes.io/projected/bd3bd8d9-b081-48bd-be29-582afe6f82fd-kube-api-access-d7ms9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm\" (UID: \"bd3bd8d9-b081-48bd-be29-582afe6f82fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" Nov 02 13:57:03.916338 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.916322 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bwcd\" (UniqueName: \"kubernetes.io/projected/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-kube-api-access-8bwcd\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:03.920696 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.920674 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fr7m\" (UniqueName: \"kubernetes.io/projected/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-kube-api-access-9fr7m\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:03.927604 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.927568 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mbpqq" Nov 02 13:57:03.934033 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.934019 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-frx4v" Nov 02 13:57:03.941205 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.941190 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-clfrl" Nov 02 13:57:03.947901 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.947699 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" Nov 02 13:57:03.983250 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:03.981935 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" Nov 02 13:57:04.000753 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.000635 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:04.007405 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.007299 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:04.025146 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.025114 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xrfvm" Nov 02 13:57:04.035248 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.035181 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ljkcn" Nov 02 13:57:04.045818 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.045602 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f9kv5" Nov 02 13:57:04.209933 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.209851 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:57:04.209933 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.209873 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:57:04.209933 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.209873 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:57:04.240126 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.230426 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-sczcr\"" Nov 02 13:57:04.240126 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.239734 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-8ms52\"" Nov 02 13:57:04.240126 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.240080 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xm2nq\"" Nov 02 13:57:04.240126 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.240090 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 13:57:04.240126 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.240094 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 13:57:04.255194 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.255168 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-frx4v"] Nov 02 13:57:04.258689 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.258672 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-clfrl"] Nov 02 13:57:04.258763 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.258722 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mbpqq"] Nov 02 13:57:04.261109 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:57:04.261084 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7157562b_09b1_44a1_95f8_f29c7e62e475.slice/crio-65ca1ce707fd022e41f01da95e2586255cdbb20ce52e377495cf999339870206 WatchSource:0}: Error finding container 65ca1ce707fd022e41f01da95e2586255cdbb20ce52e377495cf999339870206: Status 404 returned error can't find the container with id 65ca1ce707fd022e41f01da95e2586255cdbb20ce52e377495cf999339870206 Nov 02 13:57:04.264059 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:57:04.264036 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd445fa88_f52d_460f_b89a_61de9cc8d541.slice/crio-7aa1f6c8dc5c77d002e7adc5ae0cd193751861e5fa5720d72a8050a132594182 WatchSource:0}: Error finding container 7aa1f6c8dc5c77d002e7adc5ae0cd193751861e5fa5720d72a8050a132594182: Status 404 returned error can't find the container with id 7aa1f6c8dc5c77d002e7adc5ae0cd193751861e5fa5720d72a8050a132594182 Nov 02 13:57:04.282401 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.282375 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5"] Nov 02 13:57:04.284730 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:57:04.284703 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0287f6_d102_43a4_80cd_62b3acfb3991.slice/crio-ce1cb2f03a704bbc689af4e8c45aaf8ea1aa63e2bfb6ceee4c31276d2991b869 WatchSource:0}: Error finding container ce1cb2f03a704bbc689af4e8c45aaf8ea1aa63e2bfb6ceee4c31276d2991b869: Status 404 returned error can't find the container with id ce1cb2f03a704bbc689af4e8c45aaf8ea1aa63e2bfb6ceee4c31276d2991b869 Nov 02 13:57:04.321175 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.320351 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm"] Nov 02 13:57:04.375466 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.375442 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:04.375585 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.375494 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:04.375585 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.375533 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:04.375585 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.375578 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:04.375735 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.375611 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:04.375735 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:04.375636 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 13:57:04.375735 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:04.375658 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579f678cff-m54g4: secret "image-registry-tls" not found Nov 02 13:57:04.375735 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.375664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hvxsl\" (UID: \"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:04.375735 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.375710 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:04.375980 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.375737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:04.375980 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:04.375847 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 13:57:04.375980 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:04.375898 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs podName:87b5876f-0012-4bdf-b037-25070684476c nodeName:}" failed. No retries permitted until 2026-11-02 13:57:05.375880455 +0000 UTC m=+30.659996644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs") pod "router-default-7899fcdf65-hl87t" (UID: "87b5876f-0012-4bdf-b037-25070684476c") : secret "router-metrics-certs-default" not found Nov 02 13:57:04.376234 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:04.375985 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle podName:87b5876f-0012-4bdf-b037-25070684476c nodeName:}" failed. No retries permitted until 2026-11-02 13:57:05.375973714 +0000 UTC m=+30.660089898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle") pod "router-default-7899fcdf65-hl87t" (UID: "87b5876f-0012-4bdf-b037-25070684476c") : configmap references non-existent config key: service-ca.crt Nov 02 13:57:04.376234 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:04.375992 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 13:57:04.376234 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:04.376005 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b64d99df4-mtsvs: secret "image-registry-tls" not found Nov 02 13:57:04.376234 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:04.376041 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls podName:ebd0d655-5116-4a35-89ff-f7575026d4ea nodeName:}" failed. No retries permitted until 2026-11-02 13:57:05.376029151 +0000 UTC m=+30.660145321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls") pod "image-registry-b64d99df4-mtsvs" (UID: "ebd0d655-5116-4a35-89ff-f7575026d4ea") : secret "image-registry-tls" not found Nov 02 13:57:04.376234 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:04.376044 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 13:57:04.376234 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:04.376059 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls podName:f212522d-ef61-4b05-aee8-538433003f78 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:05.376048773 +0000 UTC m=+30.660164962 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls") pod "image-registry-579f678cff-m54g4" (UID: "f212522d-ef61-4b05-aee8-538433003f78") : secret "image-registry-tls" not found Nov 02 13:57:04.376234 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:04.376076 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert podName:cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:05.376066969 +0000 UTC m=+30.660183145 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hvxsl" (UID: "cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1") : secret "networking-console-plugin-cert" not found Nov 02 13:57:04.376234 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:04.376100 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 13:57:04.376234 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:04.376106 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 13:57:04.376234 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:04.376120 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls podName:1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d nodeName:}" failed. No retries permitted until 2026-11-02 13:57:05.376113171 +0000 UTC m=+30.660229381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4sfb8" (UID: "1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d") : secret "samples-operator-tls" not found Nov 02 13:57:04.376234 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:04.376154 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls podName:ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:05.376138383 +0000 UTC m=+30.660254577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls") pod "dns-default-cp4vx" (UID: "ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e") : secret "dns-default-metrics-tls" not found Nov 02 13:57:04.376234 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:04.376208 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 13:57:04.376594 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:04.376283 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls podName:b1259f5f-470e-4ca5-8760-37bff1735835 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:05.376264549 +0000 UTC m=+30.660380736 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4tgbq" (UID: "b1259f5f-470e-4ca5-8760-37bff1735835") : secret "cluster-monitoring-operator-tls" not found Nov 02 13:57:04.388538 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.388511 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-ljkcn"] Nov 02 13:57:04.394351 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:57:04.393650 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8133c4b_08bc_443b_aa8c_d914da38a2a5.slice/crio-560347c09c8c1b53def8f0495940d1d74732ef579275bf63c8798af7b6af880d WatchSource:0}: Error finding container 560347c09c8c1b53def8f0495940d1d74732ef579275bf63c8798af7b6af880d: Status 404 returned error can't find the container with id 560347c09c8c1b53def8f0495940d1d74732ef579275bf63c8798af7b6af880d Nov 02 13:57:04.403392 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.402917 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5hfrp"] Nov 02 13:57:04.406268 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.406246 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xrfvm"] Nov 02 13:57:04.410181 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:57:04.410088 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf94ad7e_96a0_4d35_9cb3_ad83f8234281.slice/crio-b4f09ad342d63dda53208d02749e76beee237dcc53715142c08bef2bb71ae359 WatchSource:0}: Error finding container b4f09ad342d63dda53208d02749e76beee237dcc53715142c08bef2bb71ae359: Status 404 returned error can't find the container with id b4f09ad342d63dda53208d02749e76beee237dcc53715142c08bef2bb71ae359 Nov 02 13:57:04.444177 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.444144 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f9kv5"] Nov 02 13:57:04.445185 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:57:04.445158 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7120b45c_1c5b_4c7c_863f_1902f96c00f5.slice/crio-f68cf294f74eeddd6fe06902b96d222dcaa47a0c0fae3630245bfc7079097180 WatchSource:0}: Error finding container f68cf294f74eeddd6fe06902b96d222dcaa47a0c0fae3630245bfc7079097180: Status 404 returned error can't find the container with id f68cf294f74eeddd6fe06902b96d222dcaa47a0c0fae3630245bfc7079097180 Nov 02 13:57:04.446343 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.446323 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fg7tg"] Nov 02 13:57:04.447275 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:57:04.447253 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f856415_9f37_415c_8dd5_65fc1ef08abc.slice/crio-171c2afeea1d998ee4db5371343a7f6fe76ae563a17cc3ec0057ab072845a099 WatchSource:0}: Error finding container 171c2afeea1d998ee4db5371343a7f6fe76ae563a17cc3ec0057ab072845a099: Status 404 returned error can't find the container with id 171c2afeea1d998ee4db5371343a7f6fe76ae563a17cc3ec0057ab072845a099 Nov 02 13:57:04.523858 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.523813 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f9kv5" event={"ID":"7120b45c-1c5b-4c7c-863f-1902f96c00f5","Type":"ContainerStarted","Data":"f68cf294f74eeddd6fe06902b96d222dcaa47a0c0fae3630245bfc7079097180"} Nov 02 13:57:04.525160 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.525122 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fg7tg" event={"ID":"4f856415-9f37-415c-8dd5-65fc1ef08abc","Type":"ContainerStarted","Data":"171c2afeea1d998ee4db5371343a7f6fe76ae563a17cc3ec0057ab072845a099"} Nov 02 13:57:04.526373 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.526347 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" event={"ID":"fc0287f6-d102-43a4-80cd-62b3acfb3991","Type":"ContainerStarted","Data":"ce1cb2f03a704bbc689af4e8c45aaf8ea1aa63e2bfb6ceee4c31276d2991b869"} Nov 02 13:57:04.527779 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.527750 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-frx4v" event={"ID":"a7095b0b-813f-4b60-989f-8a01e2743bbe","Type":"ContainerStarted","Data":"3bf2bdd286ab986d1a99b05cce8f31d8448e8f5f2e02750948fb123f4d1e261d"} Nov 02 13:57:04.527870 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.527797 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-frx4v" event={"ID":"a7095b0b-813f-4b60-989f-8a01e2743bbe","Type":"ContainerStarted","Data":"3882926d5baf6435a2113cc723f648db565876553a17643ac5db43eefecf0df3"} Nov 02 13:57:04.529224 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.529204 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-clfrl" event={"ID":"7157562b-09b1-44a1-95f8-f29c7e62e475","Type":"ContainerStarted","Data":"2a036dec49b1b74d72dc9397781bbe649fdf7a1e233a659aea4b6a49e93422cb"} Nov 02 13:57:04.529307 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.529251 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-clfrl" event={"ID":"7157562b-09b1-44a1-95f8-f29c7e62e475","Type":"ContainerStarted","Data":"65ca1ce707fd022e41f01da95e2586255cdbb20ce52e377495cf999339870206"} Nov 02 13:57:04.530449 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.530432 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xrfvm" event={"ID":"cf94ad7e-96a0-4d35-9cb3-ad83f8234281","Type":"ContainerStarted","Data":"b4f09ad342d63dda53208d02749e76beee237dcc53715142c08bef2bb71ae359"} Nov 02 13:57:04.531491 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.531474 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ljkcn" event={"ID":"a8133c4b-08bc-443b-aa8c-d914da38a2a5","Type":"ContainerStarted","Data":"560347c09c8c1b53def8f0495940d1d74732ef579275bf63c8798af7b6af880d"} Nov 02 13:57:04.532565 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.532547 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" event={"ID":"b97f6b80-f53b-4fff-97f6-dff42fb5875f","Type":"ContainerStarted","Data":"9722f613d025a8297b61622d88571c6ea757a24b8065ffca19f05dc81b4caf91"} Nov 02 13:57:04.533625 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.533607 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" event={"ID":"bd3bd8d9-b081-48bd-be29-582afe6f82fd","Type":"ContainerStarted","Data":"1e03b2c98523c22ef1d260e8838a6b498cd21d24cd65e553a471aba0f5d25e96"} Nov 02 13:57:04.534890 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.534865 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mbpqq" event={"ID":"d445fa88-f52d-460f-b89a-61de9cc8d541","Type":"ContainerStarted","Data":"869f0e8209841f9d2f116bb2ee2929ea99e042691b47c525fd25f5b523bcc0e0"} Nov 02 13:57:04.534963 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.534900 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mbpqq" event={"ID":"d445fa88-f52d-460f-b89a-61de9cc8d541","Type":"ContainerStarted","Data":"7aa1f6c8dc5c77d002e7adc5ae0cd193751861e5fa5720d72a8050a132594182"} Nov 02 13:57:04.571734 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.571662 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-frx4v" podStartSLOduration=294.571642803 podStartE2EDuration="4m54.571642803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:04.569442507 +0000 UTC m=+29.853558699" watchObservedRunningTime="2026-11-02 13:57:04.571642803 +0000 UTC m=+29.855758996" Nov 02 13:57:04.610864 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.610701 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-mbpqq" podStartSLOduration=294.610684579 podStartE2EDuration="4m54.610684579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:04.61022706 +0000 UTC m=+29.894343253" watchObservedRunningTime="2026-11-02 13:57:04.610684579 +0000 UTC m=+29.894800772" Nov 02 13:57:04.657460 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:04.657363 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-clfrl" podStartSLOduration=294.657350164 podStartE2EDuration="4m54.657350164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:04.656232164 +0000 UTC m=+29.940348369" watchObservedRunningTime="2026-11-02 13:57:04.657350164 +0000 UTC m=+29.941466355" Nov 02 13:57:05.388429 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:05.386793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:05.388429 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:05.386856 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:05.388429 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:05.386933 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hvxsl\" (UID: \"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:05.388429 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:05.387001 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:05.388429 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:05.387031 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:05.388429 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:05.387083 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:05.388429 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:05.387132 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:05.388429 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:05.387172 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:05.388429 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:05.387314 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 13:57:05.388429 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:05.387378 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls podName:b1259f5f-470e-4ca5-8760-37bff1735835 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:07.387361197 +0000 UTC m=+32.671477384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4tgbq" (UID: "b1259f5f-470e-4ca5-8760-37bff1735835") : secret "cluster-monitoring-operator-tls" not found Nov 02 13:57:05.388429 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:05.387783 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 13:57:05.388429 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:05.387836 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls podName:1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d nodeName:}" failed. No retries permitted until 2026-11-02 13:57:07.387819598 +0000 UTC m=+32.671935770 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4sfb8" (UID: "1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d") : secret "samples-operator-tls" not found Nov 02 13:57:05.388429 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:05.387899 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 13:57:05.388429 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:05.387910 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b64d99df4-mtsvs: secret "image-registry-tls" not found Nov 02 13:57:05.388429 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:05.387959 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls podName:ebd0d655-5116-4a35-89ff-f7575026d4ea nodeName:}" failed. No retries permitted until 2026-11-02 13:57:07.387932869 +0000 UTC m=+32.672049041 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls") pod "image-registry-b64d99df4-mtsvs" (UID: "ebd0d655-5116-4a35-89ff-f7575026d4ea") : secret "image-registry-tls" not found Nov 02 13:57:05.388429 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:05.388009 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 13:57:05.389325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:05.388041 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert podName:cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:07.388031653 +0000 UTC m=+32.672147823 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hvxsl" (UID: "cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1") : secret "networking-console-plugin-cert" not found Nov 02 13:57:05.389325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:05.388095 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 13:57:05.389325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:05.388125 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls podName:ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:07.388115017 +0000 UTC m=+32.672231192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls") pod "dns-default-cp4vx" (UID: "ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e") : secret "dns-default-metrics-tls" not found Nov 02 13:57:05.389325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:05.388180 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 13:57:05.389325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:05.388210 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs podName:87b5876f-0012-4bdf-b037-25070684476c nodeName:}" failed. No retries permitted until 2026-11-02 13:57:07.388200824 +0000 UTC m=+32.672317000 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs") pod "router-default-7899fcdf65-hl87t" (UID: "87b5876f-0012-4bdf-b037-25070684476c") : secret "router-metrics-certs-default" not found Nov 02 13:57:05.389325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:05.388265 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 13:57:05.389325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:05.388275 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579f678cff-m54g4: secret "image-registry-tls" not found Nov 02 13:57:05.389325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:05.388303 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls podName:f212522d-ef61-4b05-aee8-538433003f78 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:07.388293742 +0000 UTC m=+32.672409915 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls") pod "image-registry-579f678cff-m54g4" (UID: "f212522d-ef61-4b05-aee8-538433003f78") : secret "image-registry-tls" not found Nov 02 13:57:05.389325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:05.388365 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle podName:87b5876f-0012-4bdf-b037-25070684476c nodeName:}" failed. No retries permitted until 2026-11-02 13:57:07.388355986 +0000 UTC m=+32.672472157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle") pod "router-default-7899fcdf65-hl87t" (UID: "87b5876f-0012-4bdf-b037-25070684476c") : configmap references non-existent config key: service-ca.crt Nov 02 13:57:07.412453 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:07.412405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:07.412885 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:07.412486 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:07.412885 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:07.412523 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:07.412885 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:07.412577 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hvxsl\" (UID: \"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:07.412885 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:07.412621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:07.412885 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:07.412648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:07.412885 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:07.412697 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:07.412885 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:07.412748 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:07.413192 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:07.413104 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 13:57:07.413192 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:07.413131 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 13:57:07.413192 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:07.413158 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 13:57:07.413192 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:07.413180 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls podName:ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:11.413159244 +0000 UTC m=+36.697275435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls") pod "dns-default-cp4vx" (UID: "ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e") : secret "dns-default-metrics-tls" not found Nov 02 13:57:07.413325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:07.413201 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 13:57:07.413325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:07.413210 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b64d99df4-mtsvs: secret "image-registry-tls" not found Nov 02 13:57:07.413325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:07.413222 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 13:57:07.413325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:07.413201 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert podName:cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:11.413189604 +0000 UTC m=+36.697305776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hvxsl" (UID: "cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1") : secret "networking-console-plugin-cert" not found Nov 02 13:57:07.413325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:07.413248 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls podName:1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d nodeName:}" failed. No retries permitted until 2026-11-02 13:57:11.413232643 +0000 UTC m=+36.697348813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4sfb8" (UID: "1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d") : secret "samples-operator-tls" not found Nov 02 13:57:07.413325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:07.413104 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 13:57:07.413325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:07.413299 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 13:57:07.413325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:07.413263 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls podName:ebd0d655-5116-4a35-89ff-f7575026d4ea nodeName:}" failed. No retries permitted until 2026-11-02 13:57:11.413254886 +0000 UTC m=+36.697371056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls") pod "image-registry-b64d99df4-mtsvs" (UID: "ebd0d655-5116-4a35-89ff-f7575026d4ea") : secret "image-registry-tls" not found Nov 02 13:57:07.413325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:07.413313 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579f678cff-m54g4: secret "image-registry-tls" not found Nov 02 13:57:07.413325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:07.413331 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls podName:b1259f5f-470e-4ca5-8760-37bff1735835 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:11.413316357 +0000 UTC m=+36.697432540 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4tgbq" (UID: "b1259f5f-470e-4ca5-8760-37bff1735835") : secret "cluster-monitoring-operator-tls" not found Nov 02 13:57:07.413766 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:07.413350 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls podName:f212522d-ef61-4b05-aee8-538433003f78 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:11.413338111 +0000 UTC m=+36.697454308 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls") pod "image-registry-579f678cff-m54g4" (UID: "f212522d-ef61-4b05-aee8-538433003f78") : secret "image-registry-tls" not found Nov 02 13:57:07.413766 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:07.413365 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs podName:87b5876f-0012-4bdf-b037-25070684476c nodeName:}" failed. No retries permitted until 2026-11-02 13:57:11.413356763 +0000 UTC m=+36.697472936 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs") pod "router-default-7899fcdf65-hl87t" (UID: "87b5876f-0012-4bdf-b037-25070684476c") : secret "router-metrics-certs-default" not found Nov 02 13:57:07.413766 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:07.413379 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle podName:87b5876f-0012-4bdf-b037-25070684476c nodeName:}" failed. No retries permitted until 2026-11-02 13:57:11.413371222 +0000 UTC m=+36.697487393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle") pod "router-default-7899fcdf65-hl87t" (UID: "87b5876f-0012-4bdf-b037-25070684476c") : configmap references non-existent config key: service-ca.crt Nov 02 13:57:07.554584 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:07.554550 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xrfvm" event={"ID":"cf94ad7e-96a0-4d35-9cb3-ad83f8234281","Type":"ContainerStarted","Data":"a82e197710cfbd891a1f0c52c54cba52940988dc2774f1007975a1bae9822c67"} Nov 02 13:57:09.029719 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.029671 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:57:09.030158 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.029829 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:57:09.030158 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:09.029836 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 13:57:09.030158 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:09.029912 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs podName:4441e34f-75c1-40d0-9d06-af34fd3fbb50 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:41.029896507 +0000 UTC m=+66.314012677 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs") pod "network-metrics-daemon-r5vdt" (UID: "4441e34f-75c1-40d0-9d06-af34fd3fbb50") : secret "metrics-daemon-secret" not found Nov 02 13:57:09.033156 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.033128 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:57:09.072876 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.072835 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:57:09.130790 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.130757 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q67r\" (UniqueName: \"kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r\") pod \"network-check-target-pstlx\" (UID: \"7f34d424-89e7-4881-bd4f-8e77566169ed\") " pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:57:09.133171 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.133145 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q67r\" (UniqueName: \"kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r\") pod \"network-check-target-pstlx\" (UID: \"7f34d424-89e7-4881-bd4f-8e77566169ed\") " pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:57:09.353802 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.353732 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:57:09.563695 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.563651 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" event={"ID":"b97f6b80-f53b-4fff-97f6-dff42fb5875f","Type":"ContainerStarted","Data":"da77516fe6e21abfe24ed2c463d351c0d7ae87996b0679ec54bcf00482b26dd0"} Nov 02 13:57:09.563917 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.563864 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:09.565280 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.565253 2577 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-5hfrp container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused" start-of-body= Nov 02 13:57:09.565425 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.565302 2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" podUID="b97f6b80-f53b-4fff-97f6-dff42fb5875f" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused" Nov 02 13:57:09.566153 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.566125 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" event={"ID":"bd3bd8d9-b081-48bd-be29-582afe6f82fd","Type":"ContainerStarted","Data":"67d26bae59a994616fbc2d460aa9ee0a8042ac478a497572196838e94da65966"} Nov 02 13:57:09.567772 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.567751 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fg7tg" event={"ID":"4f856415-9f37-415c-8dd5-65fc1ef08abc","Type":"ContainerStarted","Data":"7951888861c1f32310bc8a232a5ab2e557ce8d3f55ed43771df23a2b98cc7fe2"} Nov 02 13:57:09.569579 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.569558 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" event={"ID":"fc0287f6-d102-43a4-80cd-62b3acfb3991","Type":"ContainerStarted","Data":"35927388778ae1f3abd2254eb630d4e33699f4d07251b4c41f446dcdafef0de3"} Nov 02 13:57:09.608216 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.608114 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xrfvm" podStartSLOduration=262.244522815 podStartE2EDuration="4m24.60809933s" totalImagesPullingTime="2.363576515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:07.599618574 +0000 UTC m=+32.883734762" watchObservedRunningTime="2026-11-02 13:57:09.60809933 +0000 UTC m=+34.892215536" Nov 02 13:57:09.608661 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.608603 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" podStartSLOduration=262.639662272 podStartE2EDuration="4m27.608591148s" totalImagesPullingTime="4.968928876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:09.607622692 +0000 UTC m=+34.891738887" watchObservedRunningTime="2026-11-02 13:57:09.608591148 +0000 UTC m=+34.892707340" Nov 02 13:57:09.652062 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.652011 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" podStartSLOduration=266.596649463 podStartE2EDuration="4m31.651996501s" totalImagesPullingTime="5.055347038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:09.650570902 +0000 UTC m=+34.934687096" watchObservedRunningTime="2026-11-02 13:57:09.651996501 +0000 UTC m=+34.936112702" Nov 02 13:57:09.679433 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.679399 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kv25k"] Nov 02 13:57:09.693238 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.693212 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pstlx"] Nov 02 13:57:09.701622 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.701580 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-fg7tg" podStartSLOduration=264.77162897 podStartE2EDuration="4m29.701563526s" totalImagesPullingTime="4.929934556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:09.70019794 +0000 UTC m=+34.984314135" watchObservedRunningTime="2026-11-02 13:57:09.701563526 +0000 UTC m=+34.985679719" Nov 02 13:57:09.746986 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:09.746908 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" podStartSLOduration=267.660857183 podStartE2EDuration="4m32.746888312s" totalImagesPullingTime="5.086031129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:09.745162046 +0000 UTC m=+35.029278241" watchObservedRunningTime="2026-11-02 13:57:09.746888312 +0000 UTC m=+35.031004509" Nov 02 13:57:09.890708 ip-10-0-134-99 kubenswrapper[2577]: W1102 13:57:09.890632 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f34d424_89e7_4881_bd4f_8e77566169ed.slice/crio-9cfe80ec146e7a05839f351a94d77561ed3198078fbd13f120a554fd3cf8f94b WatchSource:0}: Error finding container 9cfe80ec146e7a05839f351a94d77561ed3198078fbd13f120a554fd3cf8f94b: Status 404 returned error can't find the container with id 9cfe80ec146e7a05839f351a94d77561ed3198078fbd13f120a554fd3cf8f94b Nov 02 13:57:10.276852 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.276816 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x6kpv"] Nov 02 13:57:10.293643 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.293615 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:10.310259 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.310226 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x6kpv"] Nov 02 13:57:10.320618 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.320593 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 13:57:10.321898 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.321867 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 13:57:10.322216 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.322196 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xjjmq\"" Nov 02 13:57:10.350052 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.350019 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m66q\" (UniqueName: \"kubernetes.io/projected/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-api-access-6m66q\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:10.350210 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.350060 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:10.350210 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.350160 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:10.350210 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.350206 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-crio-socket\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:10.350376 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.350355 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-data-volume\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:10.452517 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.451367 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-crio-socket\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:10.452517 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.451479 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-data-volume\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:10.452517 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.451545 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m66q\" (UniqueName: \"kubernetes.io/projected/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-api-access-6m66q\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:10.452517 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.451575 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:10.452517 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.451624 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:10.452517 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:10.451722 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 13:57:10.452517 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:10.451777 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-rbac-proxy-cm podName:d32c74f9-dc27-4e9f-a20b-da4c3704acc1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:10.951761071 +0000 UTC m=+36.235877256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x6kpv" (UID: "d32c74f9-dc27-4e9f-a20b-da4c3704acc1") : configmap "kube-rbac-proxy" not found Nov 02 13:57:10.452517 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.451848 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-crio-socket\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:10.452517 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.452151 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-data-volume\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:10.453300 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:10.453216 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 13:57:10.453300 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:10.453282 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls podName:d32c74f9-dc27-4e9f-a20b-da4c3704acc1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:10.953256894 +0000 UTC m=+36.237373080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x6kpv" (UID: "d32c74f9-dc27-4e9f-a20b-da4c3704acc1") : secret "insights-runtime-extractor-tls" not found Nov 02 13:57:10.483197 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.483166 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m66q\" (UniqueName: \"kubernetes.io/projected/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-api-access-6m66q\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:10.574204 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.574166 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kv25k" event={"ID":"d1e29491-8fb3-4918-b930-796653f412ec","Type":"ContainerStarted","Data":"43bb8fb2f76dfdbf37803a8ef02b969a1fddc76b51ab28966c628e4095c4db2b"} Nov 02 13:57:10.576310 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.576279 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pstlx" event={"ID":"7f34d424-89e7-4881-bd4f-8e77566169ed","Type":"ContainerStarted","Data":"0ad2db434604fc5518539559a7ee1e7a67a9730141380f64a08440c598d0ae1a"} Nov 02 13:57:10.576424 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.576328 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pstlx" event={"ID":"7f34d424-89e7-4881-bd4f-8e77566169ed","Type":"ContainerStarted","Data":"9cfe80ec146e7a05839f351a94d77561ed3198078fbd13f120a554fd3cf8f94b"} Nov 02 13:57:10.576531 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.576510 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:57:10.578574 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.578525 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ljkcn" event={"ID":"a8133c4b-08bc-443b-aa8c-d914da38a2a5","Type":"ContainerStarted","Data":"9de7559ed2fb61f6011bc5e3dd0ad8e3d683002476712e193c93196f4923d146"} Nov 02 13:57:10.580085 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.580058 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/0.log" Nov 02 13:57:10.580181 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.580093 2577 generic.go:356] "Generic (PLEG): container finished" podID="b97f6b80-f53b-4fff-97f6-dff42fb5875f" containerID="da77516fe6e21abfe24ed2c463d351c0d7ae87996b0679ec54bcf00482b26dd0" exitCode=255 Nov 02 13:57:10.580181 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.580120 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" event={"ID":"b97f6b80-f53b-4fff-97f6-dff42fb5875f","Type":"ContainerDied","Data":"da77516fe6e21abfe24ed2c463d351c0d7ae87996b0679ec54bcf00482b26dd0"} Nov 02 13:57:10.580402 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.580383 2577 scope.go:122] "RemoveContainer" containerID="da77516fe6e21abfe24ed2c463d351c0d7ae87996b0679ec54bcf00482b26dd0" Nov 02 13:57:10.581736 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.581715 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f9kv5" event={"ID":"7120b45c-1c5b-4c7c-863f-1902f96c00f5","Type":"ContainerStarted","Data":"a14ee828ebc933ef2b148be67723e8d8e8a7a4d310f9b09dd10d211658326aeb"} Nov 02 13:57:10.658083 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.657166 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pstlx" podStartSLOduration=35.154960134 podStartE2EDuration="35.657149782s" totalImagesPullingTime="502.189648ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:10.656177274 +0000 UTC m=+35.940293469" watchObservedRunningTime="2026-11-02 13:57:10.657149782 +0000 UTC m=+35.941265974" Nov 02 13:57:10.724693 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.721770 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-f9kv5" podStartSLOduration=2.229986928 podStartE2EDuration="7.721751932s" totalImagesPullingTime="5.491765004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:10.72127341 +0000 UTC m=+36.005389603" watchObservedRunningTime="2026-11-02 13:57:10.721751932 +0000 UTC m=+36.005868122" Nov 02 13:57:10.791285 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.791226 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ljkcn" podStartSLOduration=259.243590149 podStartE2EDuration="4m24.79120542s" totalImagesPullingTime="5.547615271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:10.78779688 +0000 UTC m=+36.071913072" watchObservedRunningTime="2026-11-02 13:57:10.79120542 +0000 UTC m=+36.075321613" Nov 02 13:57:10.956858 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.956818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:10.957056 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:10.956980 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 13:57:10.957146 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:10.957058 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-rbac-proxy-cm podName:d32c74f9-dc27-4e9f-a20b-da4c3704acc1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:11.95703984 +0000 UTC m=+37.241156026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x6kpv" (UID: "d32c74f9-dc27-4e9f-a20b-da4c3704acc1") : configmap "kube-rbac-proxy" not found Nov 02 13:57:10.957300 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:10.957164 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:10.957300 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:10.957286 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 13:57:10.957396 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:10.957333 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls podName:d32c74f9-dc27-4e9f-a20b-da4c3704acc1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:11.957319951 +0000 UTC m=+37.241436135 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x6kpv" (UID: "d32c74f9-dc27-4e9f-a20b-da4c3704acc1") : secret "insights-runtime-extractor-tls" not found Nov 02 13:57:11.463591 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:11.463547 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:11.464047 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:11.463638 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hvxsl\" (UID: \"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:11.464047 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:11.463676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:11.464047 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:11.463695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:11.464047 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.463719 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 13:57:11.464047 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:11.463732 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:11.464047 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.463743 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b64d99df4-mtsvs: secret "image-registry-tls" not found Nov 02 13:57:11.464047 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.463803 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 13:57:11.464047 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.463808 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 13:57:11.464047 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.463821 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 13:57:11.464047 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.463834 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579f678cff-m54g4: secret "image-registry-tls" not found Nov 02 13:57:11.464047 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.463807 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls podName:ebd0d655-5116-4a35-89ff-f7575026d4ea nodeName:}" failed. No retries permitted until 2026-11-02 13:57:19.463789092 +0000 UTC m=+44.747905276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls") pod "image-registry-b64d99df4-mtsvs" (UID: "ebd0d655-5116-4a35-89ff-f7575026d4ea") : secret "image-registry-tls" not found Nov 02 13:57:11.464047 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.463807 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 13:57:11.464047 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:11.463870 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:11.464047 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.463897 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls podName:ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:19.463881586 +0000 UTC m=+44.747997772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls") pod "dns-default-cp4vx" (UID: "ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e") : secret "dns-default-metrics-tls" not found Nov 02 13:57:11.464047 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:11.463921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:11.464047 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.463930 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle podName:87b5876f-0012-4bdf-b037-25070684476c nodeName:}" failed. No retries permitted until 2026-11-02 13:57:19.463917891 +0000 UTC m=+44.748034075 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle") pod "router-default-7899fcdf65-hl87t" (UID: "87b5876f-0012-4bdf-b037-25070684476c") : configmap references non-existent config key: service-ca.crt Nov 02 13:57:11.464047 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.463972 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert podName:cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:19.463942324 +0000 UTC m=+44.748058520 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hvxsl" (UID: "cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1") : secret "networking-console-plugin-cert" not found Nov 02 13:57:11.464900 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.463993 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls podName:f212522d-ef61-4b05-aee8-538433003f78 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:19.463983749 +0000 UTC m=+44.748099923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls") pod "image-registry-579f678cff-m54g4" (UID: "f212522d-ef61-4b05-aee8-538433003f78") : secret "image-registry-tls" not found Nov 02 13:57:11.464900 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.464013 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 13:57:11.464900 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.464016 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs podName:87b5876f-0012-4bdf-b037-25070684476c nodeName:}" failed. No retries permitted until 2026-11-02 13:57:19.464007863 +0000 UTC m=+44.748124048 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs") pod "router-default-7899fcdf65-hl87t" (UID: "87b5876f-0012-4bdf-b037-25070684476c") : secret "router-metrics-certs-default" not found Nov 02 13:57:11.464900 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.464082 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls podName:b1259f5f-470e-4ca5-8760-37bff1735835 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:19.464067858 +0000 UTC m=+44.748184028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4tgbq" (UID: "b1259f5f-470e-4ca5-8760-37bff1735835") : secret "cluster-monitoring-operator-tls" not found Nov 02 13:57:11.464900 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:11.464142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:11.464900 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.464249 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 13:57:11.464900 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.464297 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls podName:1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d nodeName:}" failed. No retries permitted until 2026-11-02 13:57:19.464284517 +0000 UTC m=+44.748400691 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4sfb8" (UID: "1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d") : secret "samples-operator-tls" not found Nov 02 13:57:11.587230 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:11.587198 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/1.log" Nov 02 13:57:11.587671 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:11.587647 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/0.log" Nov 02 13:57:11.587771 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:11.587689 2577 generic.go:356] "Generic (PLEG): container finished" podID="b97f6b80-f53b-4fff-97f6-dff42fb5875f" containerID="f892c402188f1cfca2d6a95a0b536190c1ff017b4bf04c09813676621b9ec58d" exitCode=255 Nov 02 13:57:11.587833 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:11.587796 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" event={"ID":"b97f6b80-f53b-4fff-97f6-dff42fb5875f","Type":"ContainerDied","Data":"f892c402188f1cfca2d6a95a0b536190c1ff017b4bf04c09813676621b9ec58d"} Nov 02 13:57:11.587874 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:11.587848 2577 scope.go:122] "RemoveContainer" containerID="da77516fe6e21abfe24ed2c463d351c0d7ae87996b0679ec54bcf00482b26dd0" Nov 02 13:57:11.588077 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:11.588055 2577 scope.go:122] "RemoveContainer" containerID="f892c402188f1cfca2d6a95a0b536190c1ff017b4bf04c09813676621b9ec58d" Nov 02 13:57:11.588325 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.588301 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5hfrp_openshift-console-operator(b97f6b80-f53b-4fff-97f6-dff42fb5875f)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" podUID="b97f6b80-f53b-4fff-97f6-dff42fb5875f" Nov 02 13:57:11.970418 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:11.970380 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:11.970580 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:11.970440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:11.970580 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.970557 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 13:57:11.970671 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.970598 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 13:57:11.970671 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.970626 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls podName:d32c74f9-dc27-4e9f-a20b-da4c3704acc1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:13.9706056 +0000 UTC m=+39.254721784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x6kpv" (UID: "d32c74f9-dc27-4e9f-a20b-da4c3704acc1") : secret "insights-runtime-extractor-tls" not found Nov 02 13:57:11.970671 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:11.970660 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-rbac-proxy-cm podName:d32c74f9-dc27-4e9f-a20b-da4c3704acc1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:13.970646821 +0000 UTC m=+39.254763012 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x6kpv" (UID: "d32c74f9-dc27-4e9f-a20b-da4c3704acc1") : configmap "kube-rbac-proxy" not found Nov 02 13:57:12.592888 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:12.592859 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/1.log" Nov 02 13:57:12.593350 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:12.593278 2577 scope.go:122] "RemoveContainer" containerID="f892c402188f1cfca2d6a95a0b536190c1ff017b4bf04c09813676621b9ec58d" Nov 02 13:57:12.593494 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:12.593470 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5hfrp_openshift-console-operator(b97f6b80-f53b-4fff-97f6-dff42fb5875f)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" podUID="b97f6b80-f53b-4fff-97f6-dff42fb5875f" Nov 02 13:57:13.991998 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:13.991942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:13.992602 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:13.992018 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:13.992602 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:13.992110 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 13:57:13.992602 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:13.992149 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 13:57:13.992602 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:13.992178 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-rbac-proxy-cm podName:d32c74f9-dc27-4e9f-a20b-da4c3704acc1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:17.992159154 +0000 UTC m=+43.276275327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x6kpv" (UID: "d32c74f9-dc27-4e9f-a20b-da4c3704acc1") : configmap "kube-rbac-proxy" not found Nov 02 13:57:13.992602 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:13.992227 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls podName:d32c74f9-dc27-4e9f-a20b-da4c3704acc1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:17.992207947 +0000 UTC m=+43.276324117 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x6kpv" (UID: "d32c74f9-dc27-4e9f-a20b-da4c3704acc1") : secret "insights-runtime-extractor-tls" not found Nov 02 13:57:14.001765 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:14.001731 2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:14.002171 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:14.002156 2577 scope.go:122] "RemoveContainer" containerID="f892c402188f1cfca2d6a95a0b536190c1ff017b4bf04c09813676621b9ec58d" Nov 02 13:57:14.002369 ip-10-0-134-99 kubenswrapper[2577]: E1102 13:57:14.002341 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5hfrp_openshift-console-operator(b97f6b80-f53b-4fff-97f6-dff42fb5875f)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" podUID="b97f6b80-f53b-4fff-97f6-dff42fb5875f" Nov 02 13:57:14.601564 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:14.601531 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kv25k" event={"ID":"d1e29491-8fb3-4918-b930-796653f412ec","Type":"ContainerStarted","Data":"b2766166223f55cfc5503aa248ecbace98ab2156ae27d9f413018b53cced755b"} Nov 02 13:57:14.646511 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:14.646459 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kv25k" podStartSLOduration=35.121227484 podStartE2EDuration="39.646434138s" totalImagesPullingTime="4.525206654s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:14.64643613 +0000 UTC m=+39.930552323" watchObservedRunningTime="2026-11-02 13:57:14.646434138 +0000 UTC m=+39.930550329" Nov 02 13:57:14.760414 ip-10-0-134-99 kubenswrapper[2577]: I1102 13:57:14.760270 2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 13:57:14.760417 ip-10-0-134-99 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 13:57:14.774359 ip-10-0-134-99 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 13:57:14.774531 ip-10-0-134-99 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 13:57:14.774581 ip-10-0-134-99 systemd[1]: kubelet.service: Consumed 2.986s CPU time. Nov 02 13:57:14.784742 ip-10-0-134-99 systemd[1]: Starting Kubernetes Kubelet... Nov 02 13:57:14.855577 ip-10-0-134-99 kubenswrapper[4866]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 13:57:14.855577 ip-10-0-134-99 kubenswrapper[4866]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 13:57:14.855577 ip-10-0-134-99 kubenswrapper[4866]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 13:57:14.855577 ip-10-0-134-99 kubenswrapper[4866]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 13:57:14.857751 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857664 4866 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:57:14.857751 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857752 4866 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857756 4866 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857759 4866 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857762 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857766 4866 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857769 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857780 4866 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857783 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857786 4866 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857789 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857792 4866 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857795 4866 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857797 4866 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857800 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857802 4866 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857805 4866 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857808 4866 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857810 4866 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857813 4866 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857816 4866 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:57:14.857819 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857818 4866 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:57:14.858360 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857821 4866 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:57:14.858360 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857824 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:57:14.858360 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857827 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:57:14.858360 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857830 4866 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:57:14.858360 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857833 4866 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:57:14.858360 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857836 4866 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:57:14.858360 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857839 4866 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:57:14.858360 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857842 4866 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:57:14.858360 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857844 4866 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:57:14.858360 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857847 4866 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:57:14.858360 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857849 4866 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:57:14.858360 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857852 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:57:14.858360 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857855 4866 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:57:14.858360 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857857 4866 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:57:14.858360 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857860 4866 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:57:14.858360 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857863 4866 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:57:14.858360 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857865 4866 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:57:14.858360 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857869 4866 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:57:14.858360 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857872 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857874 4866 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857877 4866 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857879 4866 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857882 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857885 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857888 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857891 4866 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857894 4866 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857897 4866 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857899 4866 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857909 4866 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857912 4866 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857915 4866 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857918 4866 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857920 4866 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857923 4866 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857926 4866 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857928 4866 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857931 4866 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:57:14.858846 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857934 4866 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:57:14.859365 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857936 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:57:14.859365 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857939 4866 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:57:14.859365 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857942 4866 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:57:14.859365 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857958 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:57:14.859365 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857961 4866 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:57:14.859365 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857964 4866 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:57:14.859365 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857967 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:57:14.859365 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857969 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:57:14.859365 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857972 4866 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:57:14.859365 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857975 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:57:14.859365 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857978 4866 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:57:14.859365 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857988 4866 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:57:14.859365 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857992 4866 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:57:14.859365 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857995 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:57:14.859365 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.857997 4866 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:57:14.859365 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858000 4866 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:57:14.859365 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858003 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:57:14.859365 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858006 4866 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:57:14.859365 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858008 4866 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858013 4866 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858017 4866 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858021 4866 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858024 4866 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858026 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858029 4866 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858032 4866 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858035 4866 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858038 4866 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858040 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858043 4866 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858046 4866 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858049 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858051 4866 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858054 4866 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858056 4866 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858059 4866 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858061 4866 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858064 4866 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:57:14.859881 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858067 4866 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:57:14.860528 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858069 4866 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:57:14.860528 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858072 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:57:14.860528 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858075 4866 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:57:14.860528 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858077 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:57:14.860528 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858080 4866 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:57:14.860528 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858091 4866 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:57:14.860528 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858919 4866 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:57:14.860528 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858929 4866 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:57:14.860528 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858934 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:57:14.860528 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858938 4866 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:57:14.860528 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858941 4866 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:57:14.860528 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858957 4866 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:57:14.860528 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858961 4866 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:57:14.860528 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858965 4866 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:57:14.860528 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858967 4866 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:57:14.860528 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858970 4866 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:57:14.860528 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858973 4866 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:57:14.860528 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858976 4866 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:57:14.860528 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858978 4866 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858981 4866 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858984 4866 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858986 4866 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858989 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858992 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858995 4866 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.858997 4866 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859000 4866 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859003 4866 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859006 4866 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859008 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859011 4866 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859014 4866 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859017 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859020 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859022 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859025 4866 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859028 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859031 4866 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:57:14.861096 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859035 4866 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859037 4866 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859040 4866 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859042 4866 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859045 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859048 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859050 4866 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859052 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859055 4866 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859058 4866 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859060 4866 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859063 4866 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859065 4866 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859068 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859070 4866 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859073 4866 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859076 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859079 4866 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859082 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859084 4866 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:57:14.861599 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859087 4866 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:57:14.862117 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859089 4866 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:57:14.862117 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859092 4866 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:57:14.862117 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859094 4866 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:57:14.862117 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859097 4866 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:57:14.862117 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859099 4866 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:57:14.862117 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859102 4866 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:57:14.862117 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859105 4866 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:57:14.862117 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859108 4866 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:57:14.862117 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859111 4866 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:57:14.862117 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859113 4866 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:57:14.862117 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859116 4866 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:57:14.862117 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859119 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:57:14.862117 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859122 4866 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:57:14.862117 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859124 4866 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:57:14.862117 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859127 4866 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:57:14.862117 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859130 4866 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:57:14.862117 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859132 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:57:14.862117 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859135 4866 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:57:14.862117 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859137 4866 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859140 4866 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859142 4866 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859145 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859147 4866 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859150 4866 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859152 4866 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859155 4866 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859162 4866 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859165 4866 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859168 4866 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859171 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859174 4866 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859177 4866 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859180 4866 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859182 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859185 4866 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859188 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859191 4866 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859193 4866 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:57:14.862641 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859196 4866 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:57:14.863481 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859198 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:57:14.863481 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859201 4866 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:57:14.863481 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859204 4866 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:57:14.863481 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859207 4866 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:57:14.863481 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859209 4866 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:57:14.863481 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859212 4866 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:57:14.863481 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859215 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:57:14.863481 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859219 4866 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:57:14.863481 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859223 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:57:14.863481 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859226 4866 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:57:14.863481 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859230 4866 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:57:14.863481 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859233 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:57:14.863481 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859235 4866 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:57:14.863481 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859238 4866 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:57:14.863481 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859742 4866 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:57:14.863481 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859748 4866 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:57:14.863481 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859752 4866 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:57:14.863481 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859755 4866 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:57:14.863481 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859758 4866 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:57:14.864169 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859763 4866 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:57:14.864169 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859766 4866 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:57:14.864169 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859768 4866 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:57:14.864169 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859771 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:57:14.864169 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859774 4866 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:57:14.864169 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859777 4866 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:57:14.864169 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859780 4866 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:57:14.864169 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859783 4866 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:57:14.864169 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859787 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:57:14.864169 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859789 4866 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:57:14.864169 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859792 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:57:14.864169 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859795 4866 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:57:14.864169 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859797 4866 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:57:14.864169 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859800 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:57:14.864169 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859803 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:57:14.864169 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859805 4866 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:57:14.864169 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859808 4866 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:57:14.864169 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859811 4866 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:57:14.864169 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859813 4866 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859817 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859820 4866 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859823 4866 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859826 4866 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859828 4866 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859831 4866 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859833 4866 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859836 4866 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859838 4866 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859841 4866 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859843 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859846 4866 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859849 4866 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859853 4866 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859855 4866 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859858 4866 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859861 4866 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859864 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859867 4866 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859869 4866 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:57:14.864643 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859872 4866 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:57:14.865198 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859874 4866 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:57:14.865198 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859877 4866 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:57:14.865198 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859880 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:57:14.865198 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859883 4866 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:57:14.865198 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859886 4866 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:57:14.865198 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859888 4866 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:57:14.865198 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859891 4866 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:57:14.865198 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859894 4866 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:57:14.865198 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859897 4866 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:57:14.865198 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859899 4866 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:57:14.865198 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859902 4866 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:57:14.865198 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859905 4866 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:57:14.865198 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859908 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:57:14.865198 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859911 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:57:14.865198 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859914 4866 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:57:14.865198 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859917 4866 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:57:14.865198 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859920 4866 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:57:14.865198 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859922 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:57:14.865198 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859925 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:57:14.865655 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859927 4866 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:57:14.865655 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859930 4866 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:57:14.865655 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859933 4866 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:57:14.865655 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859937 4866 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:57:14.865655 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859940 4866 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:57:14.865655 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859957 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:57:14.865655 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859960 4866 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:57:14.865655 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859963 4866 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:57:14.865655 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859966 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:57:14.865655 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859969 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:57:14.865655 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859971 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:57:14.865655 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859974 4866 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:57:14.865655 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859976 4866 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:57:14.865655 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859979 4866 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:57:14.865655 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859982 4866 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:57:14.865655 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859985 4866 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:57:14.865655 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859987 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:57:14.865655 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859991 4866 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:57:14.865655 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859994 4866 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:57:14.866144 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.859998 4866 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:57:14.866144 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860001 4866 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:57:14.866144 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860003 4866 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:57:14.866144 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860006 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:57:14.866144 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860009 4866 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:57:14.866144 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860011 4866 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:57:14.866144 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860014 4866 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:57:14.866144 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860017 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:57:14.866144 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860019 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:57:14.866144 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860022 4866 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:57:14.866144 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860025 4866 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:57:14.866144 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860027 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:57:14.866144 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860030 4866 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:57:14.866144 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860033 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:57:14.866144 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860037 4866 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:57:14.866144 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860042 4866 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:57:14.866144 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860047 4866 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:57:14.866144 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860051 4866 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:57:14.866144 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860057 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860061 4866 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860065 4866 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860069 4866 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.860077 4866 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860222 4866 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860233 4866 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860242 4866 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860249 4866 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860256 4866 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860261 4866 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860268 4866 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860274 4866 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860280 4866 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860285 4866 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860291 4866 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860297 4866 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860302 4866 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860307 4866 flags.go:64] FLAG: --cgroup-root="" Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860312 4866 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860317 4866 flags.go:64] FLAG: --client-ca-file="" Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860322 4866 flags.go:64] FLAG: --cloud-provider="external" Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860328 4866 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 13:57:14.866661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860336 4866 flags.go:64] FLAG: --cluster-domain="" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860341 4866 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860346 4866 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860351 4866 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860357 4866 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860363 4866 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860368 4866 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860376 4866 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860382 4866 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860388 4866 flags.go:64] FLAG: --contention-profiling="false" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860393 4866 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860398 4866 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860404 4866 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860408 4866 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860415 4866 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860420 4866 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860425 4866 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860430 4866 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860435 4866 flags.go:64] FLAG: --enable-server="true" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860441 4866 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860447 4866 flags.go:64] FLAG: --event-burst="100" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860453 4866 flags.go:64] FLAG: --event-qps="50" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860458 4866 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860463 4866 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 13:57:14.867641 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860468 4866 flags.go:64] FLAG: --eviction-hard="" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860475 4866 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860480 4866 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860485 4866 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860490 4866 flags.go:64] FLAG: --eviction-soft="" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860495 4866 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860500 4866 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860505 4866 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860511 4866 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860516 4866 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860521 4866 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860526 4866 flags.go:64] FLAG: --feature-gates="" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860533 4866 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860538 4866 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860543 4866 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860551 4866 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860556 4866 flags.go:64] FLAG: --healthz-port="10248" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860562 4866 flags.go:64] FLAG: --help="false" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860567 4866 flags.go:64] FLAG: --hostname-override="ip-10-0-134-99.ec2.internal" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860572 4866 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860577 4866 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860582 4866 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860588 4866 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 13:57:14.868448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860593 4866 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860598 4866 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860603 4866 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860607 4866 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860613 4866 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860617 4866 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860623 4866 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860628 4866 flags.go:64] FLAG: --kube-reserved="" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860632 4866 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860636 4866 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860641 4866 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860646 4866 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860650 4866 flags.go:64] FLAG: --lock-file="" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860655 4866 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860659 4866 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860664 4866 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860673 4866 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860678 4866 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860684 4866 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860688 4866 flags.go:64] FLAG: --logging-format="text" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860694 4866 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860699 4866 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860703 4866 flags.go:64] FLAG: --manifest-url="" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860708 4866 flags.go:64] FLAG: --manifest-url-header="" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860717 4866 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 13:57:14.869031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860722 4866 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860729 4866 flags.go:64] FLAG: --max-pods="110" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860734 4866 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860739 4866 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860743 4866 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860748 4866 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860753 4866 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860758 4866 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860762 4866 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860773 4866 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860778 4866 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860783 4866 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860788 4866 flags.go:64] FLAG: --pod-cidr="" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860793 4866 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860797 4866 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860802 4866 flags.go:64] FLAG: --pods-per-core="0" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860806 4866 flags.go:64] FLAG: --port="10250" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860812 4866 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860816 4866 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-081e5f5bb815e6c3d" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860821 4866 flags.go:64] FLAG: --qos-reserved="" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860826 4866 flags.go:64] FLAG: --read-only-port="10255" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860831 4866 flags.go:64] FLAG: --register-node="true" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860837 4866 flags.go:64] FLAG: --register-with-taints="" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860843 4866 flags.go:64] FLAG: --registry-burst="10" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860847 4866 flags.go:64] FLAG: --registry-qps="5" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860852 4866 flags.go:64] FLAG: --reserved-cpus="" Nov 02 13:57:14.869718 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860858 4866 flags.go:64] FLAG: --reserved-memory="" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860864 4866 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860869 4866 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860874 4866 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860878 4866 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860885 4866 flags.go:64] FLAG: --runonce="false" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860890 4866 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860896 4866 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860900 4866 flags.go:64] FLAG: --seccomp-default="false" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860905 4866 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860910 4866 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860915 4866 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860920 4866 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860926 4866 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860930 4866 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860935 4866 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860939 4866 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860960 4866 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860966 4866 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860971 4866 flags.go:64] FLAG: --system-cgroups="" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860975 4866 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860983 4866 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860987 4866 flags.go:64] FLAG: --tls-cert-file="" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.860992 4866 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.861000 4866 flags.go:64] FLAG: --tls-min-version="" Nov 02 13:57:14.870380 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.861004 4866 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.861008 4866 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.861013 4866 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.861018 4866 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.861023 4866 flags.go:64] FLAG: --v="2" Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.861030 4866 flags.go:64] FLAG: --version="false" Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.861036 4866 flags.go:64] FLAG: --vmodule="" Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.861043 4866 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.861049 4866 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861223 4866 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861228 4866 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861232 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861237 4866 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861241 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861244 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861247 4866 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861250 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861253 4866 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861256 4866 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861259 4866 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861262 4866 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:57:14.871078 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861264 4866 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861267 4866 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861270 4866 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861272 4866 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861275 4866 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861278 4866 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861281 4866 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861283 4866 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861286 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861289 4866 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861291 4866 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861294 4866 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861297 4866 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861302 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861305 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861308 4866 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861310 4866 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861313 4866 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861316 4866 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861319 4866 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:57:14.871610 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861322 4866 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861325 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861329 4866 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861332 4866 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861336 4866 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861339 4866 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861342 4866 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861344 4866 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861347 4866 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861350 4866 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861352 4866 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861355 4866 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861358 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861360 4866 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861363 4866 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861366 4866 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861369 4866 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861371 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861374 4866 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861377 4866 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:57:14.872121 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861379 4866 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:57:14.872676 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861382 4866 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:57:14.872676 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861385 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:57:14.872676 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861387 4866 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:57:14.872676 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861390 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:57:14.872676 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861394 4866 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:57:14.872676 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861397 4866 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:57:14.872676 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861399 4866 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:57:14.872676 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861402 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:57:14.872676 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861405 4866 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:57:14.872676 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861407 4866 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:57:14.872676 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861410 4866 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:57:14.872676 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861413 4866 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:57:14.872676 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861416 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:57:14.872676 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861419 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:57:14.872676 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861422 4866 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:57:14.872676 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861424 4866 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:57:14.872676 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861427 4866 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:57:14.872676 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861429 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:57:14.872676 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861432 4866 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861435 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861437 4866 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861440 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861442 4866 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861445 4866 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861448 4866 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861450 4866 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861453 4866 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861455 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861458 4866 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861461 4866 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861464 4866 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861467 4866 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861469 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861472 4866 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861474 4866 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861477 4866 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861481 4866 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861484 4866 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:57:14.873213 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861486 4866 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:57:14.873770 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861489 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:57:14.873770 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861492 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:57:14.873770 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861495 4866 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:57:14.873770 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861497 4866 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:57:14.873770 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861500 4866 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:57:14.873770 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861503 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:57:14.873770 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861506 4866 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:57:14.873770 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861509 4866 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:57:14.873770 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861512 4866 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:57:14.873770 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861514 4866 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:57:14.873770 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861517 4866 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:57:14.873770 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861520 4866 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:57:14.873770 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861523 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:57:14.873770 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861525 4866 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:57:14.873770 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.861553 4866 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 13:57:14.874218 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861588 4866 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:57:14.874218 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861594 4866 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:57:14.874218 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861597 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:57:14.874218 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861600 4866 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:57:14.874218 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861603 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:57:14.874218 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861606 4866 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:57:14.874218 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861609 4866 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:57:14.874218 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861612 4866 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:57:14.874218 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861616 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:57:14.874218 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861619 4866 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:57:14.874218 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861621 4866 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:57:14.874218 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861624 4866 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:57:14.874218 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861627 4866 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:57:14.874218 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861629 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:57:14.874218 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861632 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:57:14.874218 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861636 4866 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:57:14.874218 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861639 4866 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:57:14.874218 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861642 4866 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:57:14.874218 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861644 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:57:14.874762 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861647 4866 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:57:14.874762 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861650 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:57:14.874762 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861653 4866 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:57:14.874762 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861655 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:57:14.874762 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861658 4866 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:57:14.874762 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861661 4866 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:57:14.874762 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861664 4866 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:57:14.874762 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861667 4866 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:57:14.874762 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861670 4866 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:57:14.874762 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861672 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:57:14.874762 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861675 4866 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:57:14.874762 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861678 4866 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:57:14.874762 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861680 4866 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:57:14.874762 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861683 4866 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:57:14.874762 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861686 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:57:14.874762 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861689 4866 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:57:14.874762 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861691 4866 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:57:14.874762 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861694 4866 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:57:14.874762 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861697 4866 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:57:14.875409 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861699 4866 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:57:14.875409 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861702 4866 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:57:14.875409 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861706 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:57:14.875409 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861709 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:57:14.875409 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861711 4866 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:57:14.875409 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861714 4866 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:57:14.875409 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861717 4866 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:57:14.875409 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861720 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:57:14.875409 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861722 4866 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:57:14.875409 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861727 4866 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:57:14.875409 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861730 4866 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:57:14.875409 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861732 4866 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:57:14.875409 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861735 4866 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:57:14.875409 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861738 4866 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:57:14.875409 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861740 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:57:14.875409 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861743 4866 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:57:14.875409 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861746 4866 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:57:14.875409 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861748 4866 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:57:14.875409 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861751 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:57:14.875902 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861756 4866 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:57:14.875902 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861760 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:57:14.875902 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861763 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:57:14.875902 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861766 4866 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:57:14.875902 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861768 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:57:14.875902 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861771 4866 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:57:14.875902 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861774 4866 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:57:14.875902 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861776 4866 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:57:14.875902 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861779 4866 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:57:14.875902 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861782 4866 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:57:14.875902 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861785 4866 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:57:14.875902 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861787 4866 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:57:14.875902 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861790 4866 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:57:14.875902 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861793 4866 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:57:14.875902 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861795 4866 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:57:14.875902 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861800 4866 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:57:14.875902 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861803 4866 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:57:14.875902 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861805 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:57:14.875902 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861808 4866 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861811 4866 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861813 4866 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861816 4866 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861820 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861823 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861826 4866 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861828 4866 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861831 4866 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861834 4866 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861836 4866 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861839 4866 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861842 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861847 4866 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861850 4866 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861853 4866 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861856 4866 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861859 4866 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861861 4866 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861864 4866 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:57:14.876542 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861866 4866 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861869 4866 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861872 4866 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861874 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861877 4866 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861880 4866 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861882 4866 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861885 4866 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861887 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861891 4866 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.861894 4866 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.866040 4866 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.866057 4866 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.866072 4866 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.866077 4866 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866157 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866163 4866 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866174 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866178 4866 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866180 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:57:14.877077 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866183 4866 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866186 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866189 4866 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866192 4866 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866195 4866 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866197 4866 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866200 4866 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866204 4866 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866208 4866 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866211 4866 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866214 4866 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866217 4866 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866220 4866 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866223 4866 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866225 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866228 4866 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866231 4866 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866233 4866 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866236 4866 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866239 4866 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:57:14.877567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866242 4866 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:57:14.878060 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866245 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:57:14.878060 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866247 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:57:14.878060 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866250 4866 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:57:14.878060 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866259 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:57:14.878060 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866261 4866 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:57:14.878060 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866264 4866 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:57:14.878060 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866267 4866 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:57:14.878060 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866269 4866 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:57:14.878060 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866272 4866 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:57:14.878060 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866275 4866 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:57:14.878060 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866277 4866 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:57:14.878060 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866280 4866 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:57:14.878060 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866282 4866 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:57:14.878060 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866285 4866 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:57:14.878060 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866287 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:57:14.878060 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866290 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:57:14.878060 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866292 4866 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:57:14.878060 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866295 4866 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:57:14.878060 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866298 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866300 4866 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866303 4866 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866306 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866308 4866 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866311 4866 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866314 4866 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866316 4866 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866319 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866322 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866325 4866 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866328 4866 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866330 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866333 4866 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866336 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866338 4866 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866341 4866 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866343 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866352 4866 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866355 4866 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:57:14.878670 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866357 4866 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:57:14.879241 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866360 4866 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:57:14.879241 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866362 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:57:14.879241 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866365 4866 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:57:14.879241 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866367 4866 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:57:14.879241 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866370 4866 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:57:14.879241 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866372 4866 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:57:14.879241 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866375 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:57:14.879241 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866379 4866 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:57:14.879241 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866382 4866 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:57:14.879241 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866386 4866 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:57:14.879241 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866389 4866 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:57:14.879241 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866392 4866 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:57:14.879241 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866395 4866 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:57:14.879241 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866398 4866 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:57:14.879241 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866401 4866 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:57:14.879241 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866405 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:57:14.879241 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866408 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:57:14.879241 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866412 4866 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:57:14.879241 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866414 4866 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:57:14.879708 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866417 4866 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:57:14.879708 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866420 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:57:14.879708 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866422 4866 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:57:14.879708 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866425 4866 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:57:14.879708 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866428 4866 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:57:14.879708 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866430 4866 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:57:14.879708 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866433 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:57:14.879708 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866436 4866 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:57:14.879708 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866438 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:57:14.879708 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866440 4866 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:57:14.879708 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866443 4866 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:57:14.879708 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866451 4866 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:57:14.879708 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866454 4866 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:57:14.879708 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866457 4866 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:57:14.879708 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866459 4866 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:57:14.879708 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866462 4866 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:57:14.879708 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866464 4866 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:57:14.879708 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866467 4866 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:57:14.879708 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866469 4866 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:57:14.880176 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866472 4866 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:57:14.880176 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866474 4866 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:57:14.880176 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.866477 4866 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:57:14.880176 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.866505 4866 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 13:57:14.880176 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867290 4866 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:57:14.880176 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867310 4866 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:57:14.880176 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867315 4866 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:57:14.880176 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867320 4866 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:57:14.880176 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867329 4866 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:57:14.880176 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867334 4866 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:57:14.880176 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867339 4866 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:57:14.880176 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867343 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:57:14.880176 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867348 4866 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:57:14.880176 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867352 4866 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:57:14.880176 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867361 4866 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:57:14.880176 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867366 4866 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867370 4866 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867375 4866 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867380 4866 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867384 4866 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867389 4866 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867393 4866 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867397 4866 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867401 4866 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867406 4866 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867436 4866 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867442 4866 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867452 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867457 4866 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867462 4866 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867466 4866 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867470 4866 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867474 4866 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867478 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867482 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:57:14.880567 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867486 4866 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:57:14.881064 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867490 4866 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:57:14.881064 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867494 4866 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:57:14.881064 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867498 4866 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:57:14.881064 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867509 4866 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:57:14.881064 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867516 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:57:14.881064 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867520 4866 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:57:14.881064 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867524 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:57:14.881064 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867528 4866 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:57:14.881064 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867532 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:57:14.881064 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867536 4866 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:57:14.881064 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867540 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:57:14.881064 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867544 4866 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:57:14.881064 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867548 4866 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:57:14.881064 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867552 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:57:14.881064 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867555 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:57:14.881064 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867560 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:57:14.881064 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867568 4866 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:57:14.881064 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867572 4866 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:57:14.881064 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867577 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:57:14.881531 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867584 4866 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:57:14.881531 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867590 4866 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:57:14.881531 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867594 4866 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:57:14.881531 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867623 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:57:14.881531 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867629 4866 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:57:14.881531 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867634 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:57:14.881531 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867639 4866 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:57:14.881531 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867643 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:57:14.881531 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867647 4866 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:57:14.881531 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867657 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:57:14.881531 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867713 4866 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:57:14.881531 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867723 4866 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:57:14.881531 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867728 4866 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:57:14.881531 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867734 4866 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:57:14.881531 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867739 4866 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:57:14.881531 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867743 4866 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:57:14.881531 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867749 4866 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:57:14.881531 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867753 4866 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:57:14.881531 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867757 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:57:14.882029 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867761 4866 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:57:14.882029 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867766 4866 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:57:14.882029 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867770 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:57:14.882029 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867780 4866 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:57:14.882029 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867785 4866 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:57:14.882029 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867791 4866 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:57:14.882029 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867798 4866 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:57:14.882029 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867803 4866 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:57:14.882029 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867808 4866 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:57:14.882029 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867812 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:57:14.882029 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867817 4866 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:57:14.882029 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867821 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:57:14.882029 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867826 4866 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:57:14.882029 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867830 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:57:14.882029 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867834 4866 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:57:14.882029 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867844 4866 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:57:14.882029 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867848 4866 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:57:14.882029 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867881 4866 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:57:14.882029 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867886 4866 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:57:14.882497 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867890 4866 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:57:14.882497 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867895 4866 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:57:14.882497 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867899 4866 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:57:14.882497 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867904 4866 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:57:14.882497 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867908 4866 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:57:14.882497 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867913 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:57:14.882497 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867917 4866 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:57:14.882497 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867922 4866 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:57:14.882497 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867926 4866 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:57:14.882497 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867936 4866 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:57:14.882497 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867940 4866 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:57:14.882497 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867963 4866 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:57:14.882497 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867968 4866 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:57:14.882497 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867973 4866 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:57:14.882497 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867977 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:57:14.882497 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867981 4866 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:57:14.882497 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.867986 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:57:14.883090 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.868046 4866 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 13:57:14.883090 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868330 4866 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:57:14.883090 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868347 4866 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:57:14.883090 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868352 4866 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:57:14.883090 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868356 4866 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:57:14.883090 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868359 4866 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:57:14.883090 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868362 4866 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:57:14.883090 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868365 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:57:14.883090 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868368 4866 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:57:14.883090 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868371 4866 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:57:14.883090 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868373 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:57:14.883090 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868377 4866 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:57:14.883090 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868379 4866 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:57:14.883090 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868384 4866 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:57:14.883090 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868389 4866 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868392 4866 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868395 4866 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868398 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868401 4866 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868404 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868407 4866 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868410 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868412 4866 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868415 4866 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868418 4866 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868421 4866 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868423 4866 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868427 4866 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868430 4866 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868433 4866 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868436 4866 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868438 4866 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868442 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868445 4866 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:57:14.883486 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868448 4866 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868451 4866 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868453 4866 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868456 4866 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868459 4866 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868462 4866 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868465 4866 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868468 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868471 4866 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868473 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868476 4866 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868480 4866 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868482 4866 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868485 4866 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868488 4866 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868490 4866 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868493 4866 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868496 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868498 4866 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868501 4866 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:57:14.884004 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868504 4866 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868506 4866 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868509 4866 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868512 4866 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868514 4866 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868518 4866 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868520 4866 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868523 4866 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868526 4866 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868528 4866 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868532 4866 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868534 4866 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868537 4866 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868542 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868544 4866 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868547 4866 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868550 4866 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868553 4866 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868556 4866 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868558 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:57:14.884541 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868561 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:57:14.885054 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868564 4866 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:57:14.885054 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868567 4866 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:57:14.885054 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868570 4866 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:57:14.885054 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868572 4866 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:57:14.885054 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868575 4866 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:57:14.885054 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868578 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:57:14.885054 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868580 4866 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:57:14.885054 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868583 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:57:14.885054 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868586 4866 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:57:14.885054 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868588 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:57:14.885054 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868591 4866 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:57:14.885054 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868594 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:57:14.885054 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868596 4866 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:57:14.885054 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868599 4866 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:57:14.885054 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868601 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:57:14.885054 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868604 4866 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:57:14.885054 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868607 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:57:14.885054 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868610 4866 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:57:14.885054 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868613 4866 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:57:14.885506 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868616 4866 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:57:14.885506 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868620 4866 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:57:14.885506 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868624 4866 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:57:14.885506 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868627 4866 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:57:14.885506 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868631 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:57:14.885506 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868634 4866 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:57:14.885506 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868636 4866 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:57:14.885506 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868639 4866 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:57:14.885506 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868642 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:57:14.885506 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868644 4866 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:57:14.885506 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868647 4866 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:57:14.885506 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868650 4866 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:57:14.885506 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:14.868653 4866 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:57:14.885506 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.868720 4866 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 13:57:14.885506 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.869737 4866 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 13:57:14.885506 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.869797 4866 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 13:57:14.885506 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.869994 4866 server.go:1014] "Starting client certificate rotation" Nov 02 13:57:14.885506 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.870142 4866 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 13:57:14.886004 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.870198 4866 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 13:51:34 +0000 UTC" deadline="2028-07-27 04:04:40.766764304 +0000 UTC" Nov 02 13:57:14.886004 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.870231 4866 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15182h7m25.896535918s" Nov 02 13:57:14.886004 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.870476 4866 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 13:57:14.886004 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.871827 4866 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 13:57:14.886004 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.873001 4866 log.go:25] "Validated CRI v1 runtime API" Nov 02 13:57:14.886004 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.874728 4866 log.go:25] "Validated CRI v1 image API" Nov 02 13:57:14.886004 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.875394 4866 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 13:57:14.886004 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.877213 4866 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 87670cfc-6288-4213-87d4-3acc6ee7e934:/dev/nvme0n1p4 f5e84f0b-4172-44de-a152-373c94270d56:/dev/nvme0n1p3] Nov 02 13:57:14.886232 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.877229 4866 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/171c2afeea1d998ee4db5371343a7f6fe76ae563a17cc3ec0057ab072845a099/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/171c2afeea1d998ee4db5371343a7f6fe76ae563a17cc3ec0057ab072845a099/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e03b2c98523c22ef1d260e8838a6b498cd21d24cd65e553a471aba0f5d25e96/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e03b2c98523c22ef1d260e8838a6b498cd21d24cd65e553a471aba0f5d25e96/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f1464fb8fc7fa5a5bd309e9d751e9661bbbc0323ec5b4aa739bb170e330599f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f1464fb8fc7fa5a5bd309e9d751e9661bbbc0323ec5b4aa739bb170e330599f/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/229933466ce509a88d22281825bfeec82aac4a7355957de74b3ca581e336a89b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/229933466ce509a88d22281825bfeec82aac4a7355957de74b3ca581e336a89b/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29fe9d1001a9638fe615caf3d6275e809f9b2ca3de1df95489bc7df1d060bb7a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29fe9d1001a9638fe615caf3d6275e809f9b2ca3de1df95489bc7df1d060bb7a/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3882926d5baf6435a2113cc723f648db565876553a17643ac5db43eefecf0df3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3882926d5baf6435a2113cc723f648db565876553a17643ac5db43eefecf0df3/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43bb8fb2f76dfdbf37803a8ef02b969a1fddc76b51ab28966c628e4095c4db2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43bb8fb2f76dfdbf37803a8ef02b969a1fddc76b51ab28966c628e4095c4db2b/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5256243e6aa938799975d72d5774db0b4f1746a1cbea1985cb18b39c80848cfa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5256243e6aa938799975d72d5774db0b4f1746a1cbea1985cb18b39c80848cfa/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/560347c09c8c1b53def8f0495940d1d74732ef579275bf63c8798af7b6af880d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/560347c09c8c1b53def8f0495940d1d74732ef579275bf63c8798af7b6af880d/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65ca1ce707fd022e41f01da95e2586255cdbb20ce52e377495cf999339870206/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65ca1ce707fd022e41f01da95e2586255cdbb20ce52e377495cf999339870206/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7aa1f6c8dc5c77d002e7adc5ae0cd193751861e5fa5720d72a8050a132594182/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7aa1f6c8dc5c77d002e7adc5ae0cd193751861e5fa5720d72a8050a132594182/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9722f613d025a8297b61622d88571c6ea757a24b8065ffca19f05dc81b4caf91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9722f613d025a8297b61622d88571c6ea757a24b8065ffca19f05dc81b4caf91/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9cfe80ec146e7a05839f351a94d77561ed3198078fbd13f120a554fd3cf8f94b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9cfe80ec146e7a05839f351a94d77561ed3198078fbd13f120a554fd3cf8f94b/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a50b491b0757cf499544e39ae2a906f296f884e34c2263e402a1659550dc730d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a50b491b0757cf499544e39ae2a906f296f884e34c2263e402a1659550dc730d/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4f09ad342d63dda53208d02749e76beee237dcc53715142c08bef2bb71ae359/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4f09ad342d63dda53208d02749e76beee237dcc53715142c08bef2bb71ae359/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bcefcdbc17296b6fd65dc27c30186c616fe97d8ee85e57cc439209990f15be02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bcefcdbc17296b6fd65dc27c30186c616fe97d8ee85e57cc439209990f15be02/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7afc641eccec919d2f64d4055b4c1403a4ce9b4f5fe903aa32bfce6199a87ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7afc641eccec919d2f64d4055b4c1403a4ce9b4f5fe903aa32bfce6199a87ba/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb03afa4c51dff071823c8c1d7f6ff6f927247e4ee5f9d1a4132d0dcb227b90c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb03afa4c51dff071823c8c1d7f6ff6f927247e4ee5f9d1a4132d0dcb227b90c/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce1cb2f03a704bbc689af4e8c45aaf8ea1aa63e2bfb6ceee4c31276d2991b869/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce1cb2f03a704bbc689af4e8c45aaf8ea1aa63e2bfb6ceee4c31276d2991b869/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d55253d3ed7a3f63aaf78c2cbc291436e2e99301c6a357735024cfe98b5ebf8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d55253d3ed7a3f63aaf78c2cbc291436e2e99301c6a357735024cfe98b5ebf8f/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e46cdba21ca0d6f183fe1812a41b196cc959e006c2c3d19b4265b7cb668e4735/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e46cdba21ca0d6f183fe1812a41b196cc959e006c2c3d19b4265b7cb668e4735/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f68cf294f74eeddd6fe06902b96d222dcaa47a0c0fae3630245bfc7079097180/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f68cf294f74eeddd6fe06902b96d222dcaa47a0c0fae3630245bfc7079097180/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/087ad47b-bcc1-4332-9706-8f0ed148252d/volumes/kubernetes.io~projected/kube-api-access-72rpv:{mountpoint:/var/lib/kubelet/pods/087ad47b-bcc1-4332-9706-8f0ed148252d/volumes/kubernetes.io~projected/kube-api-access-72rpv major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17f96927-431c-4743-8cda-eb1856feab7a/volumes/kubernetes.io~projected/kube-api-access-mr8lv:{mountpoint:/var/lib/kubelet/pods/17f96927-431c-4743-8cda-eb1856feab7a/volumes/kubernetes.io~projected/kube-api-access-mr8lv major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d/volumes/kubernetes.io~projected/kube-api-access-9fr7m:{mountpoint:/var/lib/kubelet/pods/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d/volumes/kubernetes.io~projected/kube-api-access-9fr7m major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4441e34f-75c1-40d0-9d06-af34fd3fbb50/volumes/kubernetes.io~projected/kube-api-access-vvxgk:{mountpoint:/var/lib/kubelet/pods/4441e34f-75c1-40d0-9d06-af34fd3fbb50/volumes/kubernetes.io~projected/kube-api-access-vvxgk major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/462bec67-536f-47f7-a649-860deadfa79d/volumes/kubernetes.io~projected/kube-api-access-mqtvw:{mountpoint:/var/lib/kubelet/pods/462bec67-536f-47f7-a649-860deadfa79d/volumes/kubernetes.io~projected/kube-api-access-mqtvw major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f856415-9f37-415c-8dd5-65fc1ef08abc/volumes/kubernetes.io~projected/kube-api-access-7lvl4:{mountpoint:/var/lib/kubelet/pods/4f856415-9f37-415c-8dd5-65fc1ef08abc/volumes/kubernetes.io~projected/kube-api-access-7lvl4 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f856415-9f37-415c-8dd5-65fc1ef08abc/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/4f856415-9f37-415c-8dd5-65fc1ef08abc/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7120b45c-1c5b-4c7c-863f-1902f96c00f5/volumes/kubernetes.io~projected/kube-api-access-p67v9:{mountpoint:/var/lib/kubelet/pods/7120b45c-1c5b-4c7c-863f-1902f96c00f5/volumes/kubernetes.io~projected/kube-api-access-p67v9 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7120b45c-1c5b-4c7c-863f-1902f96c00f5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7120b45c-1c5b-4c7c-863f-1902f96c00f5/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7157562b-09b1-44a1-95f8-f29c7e62e475/volumes/kubernetes.io~projected/kube-api-access-z4zbv:{mountpoint:/var/lib/kubelet/pods/7157562b-09b1-44a1-95f8-f29c7e62e475/volumes/kubernetes.io~projected/kube-api-access-z4zbv major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f34d424-89e7-4881-bd4f-8e77566169ed/volumes/kubernetes.io~projected/kube-api-access-7q67r:{mountpoint:/var/lib/kubelet/pods/7f34d424-89e7-4881-bd4f-8e77566169ed/volumes/kubernetes.io~projected/kube-api-access-7q67r major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87b5876f-0012-4bdf-b037-25070684476c/volumes/kubernetes.io~projected/kube-api-access-bw75f:{mountpoint:/var/lib/kubelet/pods/87b5876f-0012-4bdf-b037-25070684476c/volumes/kubernetes.io~projected/kube-api-access-bw75f major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87b5876f-0012-4bdf-b037-25070684476c/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/87b5876f-0012-4bdf-b037-25070684476c/volumes/kubernetes.io~secret/default-certificate major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87b5876f-0012-4bdf-b037-25070684476c/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/87b5876f-0012-4bdf-b037-25070684476c/volumes/kubernetes.io~secret/stats-auth major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7095b0b-813f-4b60-989f-8a01e2743bbe/volumes/kubernetes.io~projected/kube-api-access-trjsq:{mountpoint:/var/lib/kubelet/pods/a7095b0b-813f-4b60-989f-8a01e2743bbe/volumes/kubernetes.io~projected/kube-api-access-trjsq major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8133c4b-08bc-443b-aa8c-d914da38a2a5/volumes/kubernetes.io~projected/kube-api-access-9prcq:{mountpoint:/var/lib/kubelet/pods/a8133c4b-08bc-443b-aa8c-d914da38a2a5/volumes/kubernetes.io~projected/kube-api-access-9prcq major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa6a722d-ff18-4837-8169-f1a642a268bb/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/aa6a722d-ff18-4837-8169-f1a642a268bb/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e/volumes/kubernetes.io~projected/kube-api-access-8bwcd:{mountpoint:/var/lib/kubelet/pods/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e/volumes/kubernetes.io~projected/kube-api-access-8bwcd major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1259f5f-470e-4ca5-8760-37bff1735835/volumes/kubernetes.io~projected/kube-api-access-tht9k:{mountpoint:/var/lib/kubelet/pods/b1259f5f-470e-4ca5-8760-37bff1735835/volumes/kubernetes.io~projected/kube-api-access-tht9k major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a/volumes/kubernetes.io~projected/kube-api-access-wvxbp:{mountpoint:/var/lib/kubelet/pods/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a/volumes/kubernetes.io~projected/kube-api-access-wvxbp major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b39a3cfd-860d-48e5-bad7-68e4f2b4157a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/b39a3cfd-860d-48e5-bad7-68e4f2b4157a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b39a3cfd-860d-48e5-bad7-68e4f2b4157a/volumes/kubernetes.io~projected/kube-api-access-s78vb:{mountpoint:/var/lib/kubelet/pods/b39a3cfd-860d-48e5-bad7-68e4f2b4157a/volumes/kubernetes.io~projected/kube-api-access-s78vb major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b39a3cfd-860d-48e5-bad7-68e4f2b4157a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/b39a3cfd-860d-48e5-bad7-68e4f2b4157a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b97f6b80-f53b-4fff-97f6-dff42fb5875f/volumes/kubernetes.io~projected/kube-api-access-t5p6j:{mountpoint:/var/lib/kubelet/pods/b97f6b80-f53b-4fff-97f6-dff42fb5875f/volumes/kubernetes.io~projected/kube-api-access-t5p6j major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b97f6b80-f53b-4fff-97f6-dff42fb5875f/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b97f6b80-f53b-4fff-97f6-dff42fb5875f/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd3bd8d9-b081-48bd-be29-582afe6f82fd/volumes/kubernetes.io~projected/kube-api-access-d7ms9:{mountpoint:/var/lib/kubelet/pods/bd3bd8d9-b081-48bd-be29-582afe6f82fd/volumes/kubernetes.io~projected/kube-api-access-d7ms9 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd3bd8d9-b081-48bd-be29-582afe6f82fd/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/bd3bd8d9-b081-48bd-be29-582afe6f82fd/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf94ad7e-96a0-4d35-9cb3-ad83f8234281/volumes/kubernetes.io~projected/kube-api-access-tss59:{mountpoint:/var/lib/kubelet/pods/cf94ad7e-96a0-4d35-9cb3-ad83f8234281/volumes/kubernetes.io~projected/kube-api-access-tss59 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1e29491-8fb3-4918-b930-796653f412ec/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d1e29491-8fb3-4918-b930-796653f412ec/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d32c74f9-dc27-4e9f-a20b-da4c3704acc1/volumes/kubernetes.io~projected/kube-api-access-6m66q:{mountpoint:/var/lib/kubelet/pods/d32c74f9-dc27-4e9f-a20b-da4c3704acc1/volumes/kubernetes.io~projected/kube-api-access-6m66q major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d445fa88-f52d-460f-b89a-61de9cc8d541/volumes/kubernetes.io~projected/kube-api-access-9hbn9:{mountpoint:/var/lib/kubelet/pods/d445fa88-f52d-460f-b89a-61de9cc8d541/volumes/kubernetes.io~projected/kube-api-access-9hbn9 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9b81de7-b765-4eac-afd2-86d99495af3d/volumes/kubernetes.io~projected/kube-api-access-g6b9z:{mountpoint:/var/lib/kubelet/pods/e9b81de7-b765-4eac-afd2-86d99495af3d/volumes/kubernetes.io~projected/kube-api-access-g6b9z major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ebd0d655-5116-4a35-89ff-f7575026d4ea/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/ebd0d655-5116-4a35-89ff-f7575026d4ea/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ebd0d655-5116-4a35-89ff-f7575026d4ea/volumes/kubernetes.io~projected/kube-api-access-75khg:{mountpoint:/var/lib/kubelet/pods/ebd0d655-5116-4a35-89ff-f7575026d4ea/volumes/kubernetes.io~projected/kube-api-access-75khg major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ebd0d655-5116-4a35-89ff-f7575026d4ea/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/ebd0d655-5116-4a35-89ff-f7575026d4ea/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ebd0d655-5116-4a35-89ff-f7575026d4ea/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/ebd0d655-5116-4a35-89ff-f7575026d4ea/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f212522d-ef61-4b05-aee8-538433003f78/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/f212522d-ef61-4b05-aee8-538433003f78/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f212522d-ef61-4b05-aee8-538433003f78/volumes/kubernetes.io~projected/kube-api-access-ghtzt:{mountpoint:/var/lib/kubelet/pods/f212522d-ef61-4b05-aee8-538433003f78/volumes/kubernetes.io~projected/kube-api-access-ghtzt major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f212522d-ef61-4b05-aee8-538433003f78/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/f212522d-ef61-4b05-aee8-538433003f78/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f212522d-ef61-4b05-aee8-538433003f78/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/f212522d-ef61-4b05-aee8-538433003f78/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f57dae1b-545a-42d4-92f8-19c79e9de22d/volumes/kubernetes.io~projected/kube-api-access-qgqq8:{mountpoint:/var/lib/kubelet/pods/f57dae1b-545a-42d4-92f8-19c79e9de22d/volumes/kubernetes.io~projected/kube-api-access-qgqq8 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f651d09a-35a5-4a64-8de3-b824dcfd90ef/volumes/kubernetes.io~projected/kube-api-access-vmrvz:{mountpoint:/var/lib/kubelet/pods/f651d09a-35a5-4a64-8de3-b824dcfd90ef/volumes/kubernetes.io~projected/kube-api-access-vmrvz major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc0287f6-d102-43a4-80cd-62b3acfb3991/volumes/kubernetes.io~projected/kube-api-access-xctln:{mountpoint:/var/lib/kubelet/pods/fc0287f6-d102-43a4-80cd-62b3acfb3991/volumes/kubernetes.io~projected/kube-api-access-xctln major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc0287f6-d102-43a4-80cd-62b3acfb3991/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/fc0287f6-d102-43a4-80cd-62b3acfb3991/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e059ef6e05ec07cbc20ee98879965e750ebbb494cf3591626f46303baec270c0/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/586dd8b9ef9ff6455f19e8771229c0a97f08314ae5df7122a8dde51a8d0db522/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/a35ea4d068854382c935102b432fb9773e1f59b149ccbb27fdb51120828b9547/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/db488a4ef8a1fea1621bcd88bc6d54323a9bc2515eed30c279358b2a12838a5a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/f366ad6e4483fddee324680de17dd30fc0863737101ac7264ee8eeeceb7505af/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/3e9441fad4fe00729f08c9b9b79d8c2fa5d8de9364103f923c8646113a860aa6/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a28c50a6af63ed9b16d00d3d6f071470fcd8201d7a9a2f5e3f1d622b709fc361/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/5982343ea72fbca91ee0962167fe025511a784e40340bfc41208621cb338461e/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/7c15a4e0b5c4e5e95be6935f52c6ccbcc84999de9d8a1a23f94a5a63589d3ea8/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/f677c659ef70af3bcaf996fb95346e5ee56f35cbb3dadb8fd5c032fdff9a2814/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/e38e3accfa16d336797b08f2b2d4ace8c3f6306163c7982a9e3bd18a43f1b04b/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/5782ea327703740b5b086bb201beca108eceeb9e705d0fe0d394217872525ad5/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9fcd2ce238cdc6da74ae95a2f49460735b882652c52fe9c14d2909d25af39798/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/2ba903ca12de2343fb2f5bd9fbecce5ca5c91c61c007a31808e145b18147f39c/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9ec1dc7370bfb682addf71fddcd100af19b6a0cf5c5223869a67fcf16f1f1a7c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/999fc3a33fafa8f38957b7fd7e9ba6d0b91410d765ec1bdb8c51338c94d22668/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d6c60c45ae8e345ed6d00a2409a13bcafe1ae8186b0d87930e8db68497a63308/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/12edead52e95e73024037f0b7e8ae2208fe8ca8b000b419c7a91f24794687760/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/24f365e4ff29322442844876014c35ae6b1f920c23ae68c5db4835606dc5a8bf/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/dce3bcd0cefcd9175eba7e1a259d9453e6df857904bdca95eac14352c76fff00/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/089a477954d9ccb107ed8d40125357d3c797e0060d63815ce5b6493df900268c/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/eb915a17ab85403c989f4fff6a57a519e6643b4991acacc3733d8451237f55dc/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/2dd6ccca9db35196bea1632138a58c478146b8376821c3aae21a7ebbf5ebe490/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/f52def2f70bf78e0ac8a7d9a1ee8236e090bd3407a4b3825344507eb669240f3/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/26f3dfed544a719856299e26d97a0f2aa843f43eb45dd9dd54964740341e509a/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/f0ae27b72b2e0caf762d02cd111e15d8aa4bc096de3d9435206cf01ba3ec94ab/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/d5ded1f116d1d4ac1de626e1852139521233e5c4f9865a42b504a58694f7a366/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/a582bd27a2eadb427464e104d99e6e4656acc01770ded21b52b8e5a9168908ce/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/96695f602b3b47515491016603871b00aeb8630893d13577c6f9c043eb3fc275/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/d0f0015a568544aee496b15934cf3cccc30159758998e4751b2dfb9363702cb5/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/da2bb9b5143d4245766b2dad7282d5cb74dbea152e5b0ec2b184528d4299d5b8/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/9aa84c2bb37849581a23a9e89f44a178bbeef623d56c82431f5501f3c51627fd/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/ca6d1e4c600e8f410bec21c6604cdbd4e9ce9817a7263d18194e42804d88e24b/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/3613a939c6118c047122a71c582504ac54ba9ba091f1b884e0197723034681a1/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/2ff6256ac714d75d023d45c47ca358ae68ea1e11b4601649ef44140cad2e1be2/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/ef5c30485e348974fc76f40a417a966b79a4b1319f8b839990447b9b5f50d9be/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/20253fcbd2cd3e4d97b07d60548c47f30746747229e4dd7f0a4c581bf1982538/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/a757fc14d7d8150c47e73ecdedf05213ee55c2be207f80d4146f19ae9fd11366/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/57d20a37e852348fb9c29d79906540ab1f7a3ddb75f8b3d484ce1b5cf1af0419/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/c66aca09eb2c9a580add34271aefce4753e1475257d0979fe33806f356dfa0d4/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/d211b2e4fd8c673bf2f2063c816aedd60cfe5bdd49a40fec7f66f5c9a81d9895/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/5ecf2441ef20921fe73b9ba81d363116abb4376cb424be9177b16fc9c9a4b594/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/61e9bdc00d8993ebd579bb6855a250297ba4b57992bb5cc554afb6e3342514c8/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/907c7a076d7cb5bbca51fca682bb83073b5c8ba39a5c12592caf6fed3509bf04/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/3af0464126a7df46a16c563406d07896dfbd4d7f4b8ed0e3ea8f00b96f87995b/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/15cba65dac919dbdf9c27ef828ad4da4fc82d9c52d419a92c48cadc23a3dcb76/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/9a7a82a985d52f716e58bd01e2b8680f64d4de82a654b1fdcd554466a3a5e3b9/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/259a9469c9008af21a576905becf39736c0f5df701740b86a37122345f333ec7/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/06dd68be3275b4d75084c876a385864246c77206c88cab10fda6ebaf89196a3c/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/beaeb47dd7ab5912970a326fcf750b035cbfdcd1c5f8471a4a682ae0207023e3/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/d1a57fd3427d5773dd4411ec645547828e0eec6ee367ab2051b47e991e2862ec/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/390dbdd4d48db70420799ca7621d3b4b6eb4e8bbeb6b52a05baab1a442933897/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-95:{mountpoint:/var/lib/containers/storage/overlay/fcfe487f28cb924aab75eb7398d3346db10ea59c365bb94a5c5499534fe1a494/merged major:0 minor:95 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/189305d2a8179ffa7bd279d1585491ea99f390aebe877e2b14716f0608b11152/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 02 13:57:14.886455 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.885982 4866 manager.go:217] Machine: {Timestamp:2026-11-02 13:57:14.885505311 +0000 UTC m=+0.069352638 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100942 MemoryCapacity:33164324864 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b0e2f1ff1f39e3109a5343908f9ad SystemUUID:ec2b0e2f-1ff1-f39e-3109-a5343908f9ad BootID:8ef70e54-d339-4b9e-a488-f8cdfdc46ae1 Filesystems:[{Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29fe9d1001a9638fe615caf3d6275e809f9b2ca3de1df95489bc7df1d060bb7a/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/b97f6b80-f53b-4fff-97f6-dff42fb5875f/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f68cf294f74eeddd6fe06902b96d222dcaa47a0c0fae3630245bfc7079097180/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/087ad47b-bcc1-4332-9706-8f0ed148252d/volumes/kubernetes.io~projected/kube-api-access-72rpv DeviceMajor:0 DeviceMinor:60 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/f651d09a-35a5-4a64-8de3-b824dcfd90ef/volumes/kubernetes.io~projected/kube-api-access-vmrvz DeviceMajor:0 DeviceMinor:63 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b39a3cfd-860d-48e5-bad7-68e4f2b4157a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/229933466ce509a88d22281825bfeec82aac4a7355957de74b3ca581e336a89b/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ebd0d655-5116-4a35-89ff-f7575026d4ea/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/7120b45c-1c5b-4c7c-863f-1902f96c00f5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce1cb2f03a704bbc689af4e8c45aaf8ea1aa63e2bfb6ceee4c31276d2991b869/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/e9b81de7-b765-4eac-afd2-86d99495af3d/volumes/kubernetes.io~projected/kube-api-access-g6b9z DeviceMajor:0 DeviceMinor:62 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/f212522d-ef61-4b05-aee8-538433003f78/volumes/kubernetes.io~projected/kube-api-access-ghtzt DeviceMajor:0 DeviceMinor:215 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b39a3cfd-860d-48e5-bad7-68e4f2b4157a/volumes/kubernetes.io~projected/kube-api-access-s78vb DeviceMajor:0 DeviceMinor:61 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/bd3bd8d9-b081-48bd-be29-582afe6f82fd/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43bb8fb2f76dfdbf37803a8ef02b969a1fddc76b51ab28966c628e4095c4db2b/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87b5876f-0012-4bdf-b037-25070684476c/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:45 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/b1259f5f-470e-4ca5-8760-37bff1735835/volumes/kubernetes.io~projected/kube-api-access-tht9k DeviceMajor:0 DeviceMinor:213 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/171c2afeea1d998ee4db5371343a7f6fe76ae563a17cc3ec0057ab072845a099/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b39a3cfd-860d-48e5-bad7-68e4f2b4157a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/d32c74f9-dc27-4e9f-a20b-da4c3704acc1/volumes/kubernetes.io~projected/kube-api-access-6m66q DeviceMajor:0 DeviceMinor:335 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a/volumes/kubernetes.io~projected/kube-api-access-wvxbp DeviceMajor:0 DeviceMinor:64 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd3bd8d9-b081-48bd-be29-582afe6f82fd/volumes/kubernetes.io~projected/kube-api-access-d7ms9 DeviceMajor:0 DeviceMinor:221 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7aa1f6c8dc5c77d002e7adc5ae0cd193751861e5fa5720d72a8050a132594182/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8133c4b-08bc-443b-aa8c-d914da38a2a5/volumes/kubernetes.io~projected/kube-api-access-9prcq DeviceMajor:0 DeviceMinor:212 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/ebd0d655-5116-4a35-89ff-f7575026d4ea/volumes/kubernetes.io~projected/kube-api-access-75khg DeviceMajor:0 DeviceMinor:218 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e/volumes/kubernetes.io~projected/kube-api-access-8bwcd DeviceMajor:0 DeviceMinor:226 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d55253d3ed7a3f63aaf78c2cbc291436e2e99301c6a357735024cfe98b5ebf8f/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4f09ad342d63dda53208d02749e76beee237dcc53715142c08bef2bb71ae359/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1e29491-8fb3-4918-b930-796653f412ec/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb03afa4c51dff071823c8c1d7f6ff6f927247e4ee5f9d1a4132d0dcb227b90c/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17f96927-431c-4743-8cda-eb1856feab7a/volumes/kubernetes.io~projected/kube-api-access-mr8lv DeviceMajor:0 DeviceMinor:59 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/87b5876f-0012-4bdf-b037-25070684476c/volumes/kubernetes.io~projected/kube-api-access-bw75f DeviceMajor:0 DeviceMinor:224 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d445fa88-f52d-460f-b89a-61de9cc8d541/volumes/kubernetes.io~projected/kube-api-access-9hbn9 DeviceMajor:0 DeviceMinor:223 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/4441e34f-75c1-40d0-9d06-af34fd3fbb50/volumes/kubernetes.io~projected/kube-api-access-vvxgk DeviceMajor:0 DeviceMinor:67 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/fc0287f6-d102-43a4-80cd-62b3acfb3991/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/a7095b0b-813f-4b60-989f-8a01e2743bbe/volumes/kubernetes.io~projected/kube-api-access-trjsq DeviceMajor:0 DeviceMinor:216 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3882926d5baf6435a2113cc723f648db565876553a17643ac5db43eefecf0df3/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9cfe80ec146e7a05839f351a94d77561ed3198078fbd13f120a554fd3cf8f94b/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a50b491b0757cf499544e39ae2a906f296f884e34c2263e402a1659550dc730d/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/462bec67-536f-47f7-a649-860deadfa79d/volumes/kubernetes.io~projected/kube-api-access-mqtvw DeviceMajor:0 DeviceMinor:65 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-95 DeviceMajor:0 DeviceMinor:95 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f856415-9f37-415c-8dd5-65fc1ef08abc/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/4f856415-9f37-415c-8dd5-65fc1ef08abc/volumes/kubernetes.io~projected/kube-api-access-7lvl4 DeviceMajor:0 DeviceMinor:214 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f34d424-89e7-4881-bd4f-8e77566169ed/volumes/kubernetes.io~projected/kube-api-access-7q67r DeviceMajor:0 DeviceMinor:293 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bcefcdbc17296b6fd65dc27c30186c616fe97d8ee85e57cc439209990f15be02/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/7157562b-09b1-44a1-95f8-f29c7e62e475/volumes/kubernetes.io~projected/kube-api-access-z4zbv DeviceMajor:0 DeviceMinor:220 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d/volumes/kubernetes.io~projected/kube-api-access-9fr7m DeviceMajor:0 DeviceMinor:227 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7afc641eccec919d2f64d4055b4c1403a4ce9b4f5fe903aa32bfce6199a87ba/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ebd0d655-5116-4a35-89ff-f7575026d4ea/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:209 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/7120b45c-1c5b-4c7c-863f-1902f96c00f5/volumes/kubernetes.io~projected/kube-api-access-p67v9 DeviceMajor:0 DeviceMinor:222 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f57dae1b-545a-42d4-92f8-19c79e9de22d/volumes/kubernetes.io~projected/kube-api-access-qgqq8 DeviceMajor:0 DeviceMinor:66 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/f212522d-ef61-4b05-aee8-538433003f78/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:210 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/aa6a722d-ff18-4837-8169-f1a642a268bb/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc0287f6-d102-43a4-80cd-62b3acfb3991/volumes/kubernetes.io~projected/kube-api-access-xctln DeviceMajor:0 DeviceMinor:219 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5256243e6aa938799975d72d5774db0b4f1746a1cbea1985cb18b39c80848cfa/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f1464fb8fc7fa5a5bd309e9d751e9661bbbc0323ec5b4aa739bb170e330599f/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/b97f6b80-f53b-4fff-97f6-dff42fb5875f/volumes/kubernetes.io~projected/kube-api-access-t5p6j DeviceMajor:0 DeviceMinor:211 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65ca1ce707fd022e41f01da95e2586255cdbb20ce52e377495cf999339870206/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f212522d-ef61-4b05-aee8-538433003f78/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:206 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/560347c09c8c1b53def8f0495940d1d74732ef579275bf63c8798af7b6af880d/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f212522d-ef61-4b05-aee8-538433003f78/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/87b5876f-0012-4bdf-b037-25070684476c/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:204 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/cf94ad7e-96a0-4d35-9cb3-ad83f8234281/volumes/kubernetes.io~projected/kube-api-access-tss59 DeviceMajor:0 DeviceMinor:217 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e03b2c98523c22ef1d260e8838a6b498cd21d24cd65e553a471aba0f5d25e96/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e46cdba21ca0d6f183fe1812a41b196cc959e006c2c3d19b4265b7cb668e4735/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/ebd0d655-5116-4a35-89ff-f7575026d4ea/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9722f613d025a8297b61622d88571c6ea757a24b8065ffca19f05dc81b4caf91/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:171c2afeea1d998 MacAddress:f6:17:64:40:7d:ed Speed:10000 Mtu:8901} {Name:1e03b2c98523c22 MacAddress:96:35:27:85:8e:ca Speed:10000 Mtu:8901} {Name:3882926d5baf643 MacAddress:52:7d:45:4e:eb:10 Speed:10000 Mtu:8901} {Name:43bb8fb2f76dfdb MacAddress:a6:b2:56:b1:ae:8d Speed:10000 Mtu:8901} {Name:560347c09c8c1b5 MacAddress:9a:45:91:9d:cc:f8 Speed:10000 Mtu:8901} {Name:65ca1ce707fd022 MacAddress:22:6d:57:38:e4:52 Speed:10000 Mtu:8901} {Name:7aa1f6c8dc5c77d MacAddress:66:d7:34:12:97:0c Speed:10000 Mtu:8901} {Name:9722f613d025a82 MacAddress:86:59:77:a9:15:63 Speed:10000 Mtu:8901} {Name:9cfe80ec146e7a0 MacAddress:36:fe:fb:60:8c:27 Speed:10000 Mtu:8901} {Name:b4f09ad342d63dd MacAddress:72:3a:0e:99:39:d2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cc:14:8d:cd Speed:0 Mtu:9001} {Name:br-int MacAddress:96:cb:dd:e1:a3:9f Speed:0 Mtu:8901} {Name:ce1cb2f03a704bb MacAddress:d2:ca:c5:5d:f8:bd Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:cc:14:8d:cd Speed:0 Mtu:9001} {Name:f68cf294f74eedd MacAddress:96:77:a3:c4:d1:77 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ca:a3:c2:cb:34:c2 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:5e:e2:73:e7:7c:4a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164324864 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 13:57:14.886455 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.886313 4866 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 13:57:14.886455 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.886361 4866 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 13:57:14.886693 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.886655 4866 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 13:57:14.888080 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.886674 4866 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-99.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 13:57:14.888080 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.887028 4866 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 13:57:14.888080 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.887043 4866 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 13:57:14.888080 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.887063 4866 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 13:57:14.888080 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.887093 4866 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 13:57:14.888080 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.887123 4866 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 13:57:14.888080 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.887443 4866 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 13:57:14.888080 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.887627 4866 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 13:57:14.888080 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.887687 4866 kubelet.go:493] "Attempting to sync node with API server" Nov 02 13:57:14.888080 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.887711 4866 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 13:57:14.888080 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.887726 4866 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 13:57:14.888080 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.887746 4866 kubelet.go:405] "Adding apiserver pod source" Nov 02 13:57:14.888080 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.887758 4866 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 13:57:14.889196 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.889178 4866 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 13:57:14.889196 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.889195 4866 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 13:57:14.889759 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.889740 4866 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 13:57:14.892330 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.892317 4866 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 13:57:14.892330 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.892330 4866 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 13:57:14.892511 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.892439 4866 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 13:57:14.892665 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.892653 4866 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 13:57:14.892695 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.892684 4866 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 13:57:14.892820 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.892810 4866 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 13:57:14.892853 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.892828 4866 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 13:57:14.892853 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.892834 4866 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 13:57:14.892853 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.892840 4866 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 13:57:14.892853 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.892846 4866 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 13:57:14.892853 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.892853 4866 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 13:57:14.893035 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.892863 4866 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 13:57:14.893035 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.892870 4866 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 13:57:14.893035 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.892883 4866 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 13:57:14.893035 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.892888 4866 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 13:57:14.893035 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.892902 4866 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 13:57:14.893035 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.892911 4866 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 13:57:14.893035 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.892938 4866 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 13:57:14.893035 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.892965 4866 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 13:57:14.896379 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.896364 4866 server.go:1263] "Started kubelet" Nov 02 13:57:14.896455 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.896418 4866 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 13:57:14.896767 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.896517 4866 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 13:57:14.896767 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.896590 4866 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 13:57:14.896910 ip-10-0-134-99 systemd[1]: Started Kubernetes Kubelet. Nov 02 13:57:14.897025 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.896917 4866 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 13:57:14.898287 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.898267 4866 server.go:317] "Adding debug handlers to kubelet server" Nov 02 13:57:14.901903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.901412 4866 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 13:57:14.901903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.901655 4866 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 13:57:14.901903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.901706 4866 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:51:37 +0000 UTC" deadline="2028-04-15 11:40:19.626342435 +0000 UTC" Nov 02 13:57:14.901903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.901742 4866 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12717h43m4.724603754s" Nov 02 13:57:14.901903 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:14.901780 4866 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-99.ec2.internal\" not found" Nov 02 13:57:14.901903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.901834 4866 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 13:57:14.901903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.901851 4866 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 13:57:14.902401 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.902324 4866 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 13:57:14.905211 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.904747 4866 factory.go:55] Registering systemd factory Nov 02 13:57:14.905211 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.904770 4866 factory.go:223] Registration of the systemd container factory successfully Nov 02 13:57:14.905555 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.905531 4866 factory.go:153] Registering CRI-O factory Nov 02 13:57:14.905555 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.905557 4866 factory.go:223] Registration of the crio container factory successfully Nov 02 13:57:14.905710 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.905577 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9b81de7-b765-4eac-afd2-86d99495af3d" volumeName="kubernetes.io/configmap/e9b81de7-b765-4eac-afd2-86d99495af3d-serviceca" seLinuxMountContext="" Nov 02 13:57:14.905710 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.905621 4866 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 13:57:14.905710 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.905645 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d" volumeName="kubernetes.io/configmap/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-config" seLinuxMountContext="" Nov 02 13:57:14.905710 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.905655 4866 factory.go:103] Registering Raw factory Nov 02 13:57:14.905710 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.905661 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f856415-9f37-415c-8dd5-65fc1ef08abc" volumeName="kubernetes.io/empty-dir/4f856415-9f37-415c-8dd5-65fc1ef08abc-snapshots" seLinuxMountContext="" Nov 02 13:57:14.905710 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.905691 4866 manager.go:1199] Started watching for new ooms in manager Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.905688 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f856415-9f37-415c-8dd5-65fc1ef08abc" volumeName="kubernetes.io/projected/4f856415-9f37-415c-8dd5-65fc1ef08abc-kube-api-access-7lvl4" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906323 4866 manager.go:319] Starting recovery of all containers Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906314 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b39a3cfd-860d-48e5-bad7-68e4f2b4157a" volumeName="kubernetes.io/configmap/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-ovnkube-config" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906353 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b97f6b80-f53b-4fff-97f6-dff42fb5875f" volumeName="kubernetes.io/configmap/b97f6b80-f53b-4fff-97f6-dff42fb5875f-trusted-ca" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906374 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebd0d655-5116-4a35-89ff-f7575026d4ea" volumeName="kubernetes.io/empty-dir/ebd0d655-5116-4a35-89ff-f7575026d4ea-ca-trust-extracted" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906388 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f212522d-ef61-4b05-aee8-538433003f78" volumeName="kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-trusted-ca" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906414 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="087ad47b-bcc1-4332-9706-8f0ed148252d" volumeName="kubernetes.io/empty-dir/087ad47b-bcc1-4332-9706-8f0ed148252d-tmp-dir" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906428 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d" volumeName="kubernetes.io/projected/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-kube-api-access-9fr7m" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906449 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4441e34f-75c1-40d0-9d06-af34fd3fbb50" volumeName="kubernetes.io/projected/4441e34f-75c1-40d0-9d06-af34fd3fbb50-kube-api-access-vvxgk" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906464 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87b5876f-0012-4bdf-b037-25070684476c" volumeName="kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-stats-auth" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906480 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa6a722d-ff18-4837-8169-f1a642a268bb" volumeName="kubernetes.io/configmap/aa6a722d-ff18-4837-8169-f1a642a268bb-konnectivity-ca" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906498 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a" volumeName="kubernetes.io/empty-dir/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-tmp" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906517 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f212522d-ef61-4b05-aee8-538433003f78" volumeName="kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-kube-api-access-ghtzt" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906535 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f651d09a-35a5-4a64-8de3-b824dcfd90ef" volumeName="kubernetes.io/projected/f651d09a-35a5-4a64-8de3-b824dcfd90ef-kube-api-access-vmrvz" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906549 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="462bec67-536f-47f7-a649-860deadfa79d" volumeName="kubernetes.io/configmap/462bec67-536f-47f7-a649-860deadfa79d-cni-binary-copy" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906565 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd3bd8d9-b081-48bd-be29-582afe6f82fd" volumeName="kubernetes.io/configmap/bd3bd8d9-b081-48bd-be29-582afe6f82fd-config" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906577 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd3bd8d9-b081-48bd-be29-582afe6f82fd" volumeName="kubernetes.io/secret/bd3bd8d9-b081-48bd-be29-582afe6f82fd-serving-cert" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906594 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f856415-9f37-415c-8dd5-65fc1ef08abc" volumeName="kubernetes.io/empty-dir/4f856415-9f37-415c-8dd5-65fc1ef08abc-tmp" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906611 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7157562b-09b1-44a1-95f8-f29c7e62e475" volumeName="kubernetes.io/projected/7157562b-09b1-44a1-95f8-f29c7e62e475-kube-api-access-z4zbv" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906625 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1259f5f-470e-4ca5-8760-37bff1735835" volumeName="kubernetes.io/configmap/b1259f5f-470e-4ca5-8760-37bff1735835-telemetry-config" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906641 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b39a3cfd-860d-48e5-bad7-68e4f2b4157a" volumeName="kubernetes.io/configmap/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-env-overrides" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906653 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f212522d-ef61-4b05-aee8-538433003f78" volumeName="kubernetes.io/empty-dir/f212522d-ef61-4b05-aee8-538433003f78-ca-trust-extracted" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906665 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc0287f6-d102-43a4-80cd-62b3acfb3991" volumeName="kubernetes.io/configmap/fc0287f6-d102-43a4-80cd-62b3acfb3991-config" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906681 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc0287f6-d102-43a4-80cd-62b3acfb3991" volumeName="kubernetes.io/projected/fc0287f6-d102-43a4-80cd-62b3acfb3991-kube-api-access-xctln" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906693 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b97f6b80-f53b-4fff-97f6-dff42fb5875f" volumeName="kubernetes.io/secret/b97f6b80-f53b-4fff-97f6-dff42fb5875f-serving-cert" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906711 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebd0d655-5116-4a35-89ff-f7575026d4ea" volumeName="kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-certificates" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906738 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="462bec67-536f-47f7-a649-860deadfa79d" volumeName="kubernetes.io/configmap/462bec67-536f-47f7-a649-860deadfa79d-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906749 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f856415-9f37-415c-8dd5-65fc1ef08abc" volumeName="kubernetes.io/configmap/4f856415-9f37-415c-8dd5-65fc1ef08abc-service-ca-bundle" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906761 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f856415-9f37-415c-8dd5-65fc1ef08abc" volumeName="kubernetes.io/secret/4f856415-9f37-415c-8dd5-65fc1ef08abc-serving-cert" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906779 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87b5876f-0012-4bdf-b037-25070684476c" volumeName="kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-default-certificate" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906801 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a" volumeName="kubernetes.io/empty-dir/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-tuned" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906819 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f212522d-ef61-4b05-aee8-538433003f78" volumeName="kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-bound-sa-token" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906835 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f212522d-ef61-4b05-aee8-538433003f78" volumeName="kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-image-registry-private-configuration" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906850 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f651d09a-35a5-4a64-8de3-b824dcfd90ef" volumeName="kubernetes.io/configmap/f651d09a-35a5-4a64-8de3-b824dcfd90ef-iptables-alerter-script" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906874 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8133c4b-08bc-443b-aa8c-d914da38a2a5" volumeName="kubernetes.io/projected/a8133c4b-08bc-443b-aa8c-d914da38a2a5-kube-api-access-9prcq" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906888 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b39a3cfd-860d-48e5-bad7-68e4f2b4157a" volumeName="kubernetes.io/secret/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906906 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebd0d655-5116-4a35-89ff-f7575026d4ea" volumeName="kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-bound-sa-token" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906921 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc0287f6-d102-43a4-80cd-62b3acfb3991" volumeName="kubernetes.io/secret/fc0287f6-d102-43a4-80cd-62b3acfb3991-serving-cert" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906935 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f212522d-ef61-4b05-aee8-538433003f78" volumeName="kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-installation-pull-secrets" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906979 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1" volumeName="kubernetes.io/configmap/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-nginx-conf" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.906994 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf94ad7e-96a0-4d35-9cb3-ad83f8234281" volumeName="kubernetes.io/projected/cf94ad7e-96a0-4d35-9cb3-ad83f8234281-kube-api-access-tss59" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907012 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1e29491-8fb3-4918-b930-796653f412ec" volumeName="kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907028 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7120b45c-1c5b-4c7c-863f-1902f96c00f5" volumeName="kubernetes.io/projected/7120b45c-1c5b-4c7c-863f-1902f96c00f5-kube-api-access-p67v9" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907043 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7120b45c-1c5b-4c7c-863f-1902f96c00f5" volumeName="kubernetes.io/secret/7120b45c-1c5b-4c7c-863f-1902f96c00f5-cert" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907066 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e" volumeName="kubernetes.io/projected/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-kube-api-access-8bwcd" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907079 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f57dae1b-545a-42d4-92f8-19c79e9de22d" volumeName="kubernetes.io/projected/f57dae1b-545a-42d4-92f8-19c79e9de22d-kube-api-access-qgqq8" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907095 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa6a722d-ff18-4837-8169-f1a642a268bb" volumeName="kubernetes.io/secret/aa6a722d-ff18-4837-8169-f1a642a268bb-agent-certs" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907108 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b39a3cfd-860d-48e5-bad7-68e4f2b4157a" volumeName="kubernetes.io/projected/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-kube-api-access-s78vb" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907120 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebd0d655-5116-4a35-89ff-f7575026d4ea" volumeName="kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-image-registry-private-configuration" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907139 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="462bec67-536f-47f7-a649-860deadfa79d" volumeName="kubernetes.io/configmap/462bec67-536f-47f7-a649-860deadfa79d-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907152 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7095b0b-813f-4b60-989f-8a01e2743bbe" volumeName="kubernetes.io/projected/a7095b0b-813f-4b60-989f-8a01e2743bbe-kube-api-access-trjsq" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907170 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a" volumeName="kubernetes.io/projected/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-kube-api-access-wvxbp" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907185 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd3bd8d9-b081-48bd-be29-582afe6f82fd" volumeName="kubernetes.io/projected/bd3bd8d9-b081-48bd-be29-582afe6f82fd-kube-api-access-d7ms9" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907198 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9b81de7-b765-4eac-afd2-86d99495af3d" volumeName="kubernetes.io/projected/e9b81de7-b765-4eac-afd2-86d99495af3d-kube-api-access-g6b9z" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907239 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="087ad47b-bcc1-4332-9706-8f0ed148252d" volumeName="kubernetes.io/projected/087ad47b-bcc1-4332-9706-8f0ed148252d-kube-api-access-72rpv" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907252 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1259f5f-470e-4ca5-8760-37bff1735835" volumeName="kubernetes.io/projected/b1259f5f-470e-4ca5-8760-37bff1735835-kube-api-access-tht9k" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907274 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b39a3cfd-860d-48e5-bad7-68e4f2b4157a" volumeName="kubernetes.io/configmap/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-ovnkube-script-lib" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907288 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b97f6b80-f53b-4fff-97f6-dff42fb5875f" volumeName="kubernetes.io/projected/b97f6b80-f53b-4fff-97f6-dff42fb5875f-kube-api-access-t5p6j" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907302 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17f96927-431c-4743-8cda-eb1856feab7a" volumeName="kubernetes.io/configmap/17f96927-431c-4743-8cda-eb1856feab7a-cni-binary-copy" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907320 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17f96927-431c-4743-8cda-eb1856feab7a" volumeName="kubernetes.io/projected/17f96927-431c-4743-8cda-eb1856feab7a-kube-api-access-mr8lv" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907334 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f856415-9f37-415c-8dd5-65fc1ef08abc" volumeName="kubernetes.io/configmap/4f856415-9f37-415c-8dd5-65fc1ef08abc-trusted-ca-bundle" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907351 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87b5876f-0012-4bdf-b037-25070684476c" volumeName="kubernetes.io/projected/87b5876f-0012-4bdf-b037-25070684476c-kube-api-access-bw75f" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907365 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b97f6b80-f53b-4fff-97f6-dff42fb5875f" volumeName="kubernetes.io/configmap/b97f6b80-f53b-4fff-97f6-dff42fb5875f-config" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907377 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebd0d655-5116-4a35-89ff-f7575026d4ea" volumeName="kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-kube-api-access-75khg" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907394 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebd0d655-5116-4a35-89ff-f7575026d4ea" volumeName="kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-installation-pull-secrets" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907411 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f34d424-89e7-4881-bd4f-8e77566169ed" volumeName="kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907430 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e" volumeName="kubernetes.io/empty-dir/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-tmp-dir" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907449 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d32c74f9-dc27-4e9f-a20b-da4c3704acc1" volumeName="kubernetes.io/empty-dir/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-data-volume" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907463 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebd0d655-5116-4a35-89ff-f7575026d4ea" volumeName="kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-trusted-ca" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907480 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d445fa88-f52d-460f-b89a-61de9cc8d541" volumeName="kubernetes.io/projected/d445fa88-f52d-460f-b89a-61de9cc8d541-kube-api-access-9hbn9" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907493 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17f96927-431c-4743-8cda-eb1856feab7a" volumeName="kubernetes.io/configmap/17f96927-431c-4743-8cda-eb1856feab7a-multus-daemon-config" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907544 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="462bec67-536f-47f7-a649-860deadfa79d" volumeName="kubernetes.io/projected/462bec67-536f-47f7-a649-860deadfa79d-kube-api-access-mqtvw" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907558 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e" volumeName="kubernetes.io/configmap/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-config-volume" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907572 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d32c74f9-dc27-4e9f-a20b-da4c3704acc1" volumeName="kubernetes.io/projected/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-api-access-6m66q" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907593 4866 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f212522d-ef61-4b05-aee8-538433003f78" volumeName="kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-registry-certificates" seLinuxMountContext="" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907608 4866 reconstruct.go:97] "Volume reconstruction finished" Nov 02 13:57:14.907980 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.907615 4866 reconciler.go:29] "Reconciler: start to sync state" Nov 02 13:57:14.920610 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.920582 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/1.log" Nov 02 13:57:14.920790 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.920752 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/1.log" Nov 02 13:57:14.936499 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.936461 4866 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 13:57:14.938027 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.938003 4866 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 13:57:14.938027 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.938022 4866 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 13:57:14.938238 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.938041 4866 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 13:57:14.938238 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:14.938070 4866 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 13:57:14.950877 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.950842 4866 generic.go:356] "Generic (PLEG): container finished" podID="462bec67-536f-47f7-a649-860deadfa79d" containerID="15c20953e390f3bc35bfedc10db83555bc196297c5588a19ee0b76a890683e2e" exitCode=0 Nov 02 13:57:14.950877 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.950870 4866 generic.go:356] "Generic (PLEG): container finished" podID="462bec67-536f-47f7-a649-860deadfa79d" containerID="01c71ff8650cea18fa1f47e95e0b0a7b3d96eef65d70a5c93da5bd6eabaacf06" exitCode=0 Nov 02 13:57:14.950877 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.950878 4866 generic.go:356] "Generic (PLEG): container finished" podID="462bec67-536f-47f7-a649-860deadfa79d" containerID="995453d0632a5ae3b0f8f2218aa347596528f9c34b62737530b10c651eefa926" exitCode=0 Nov 02 13:57:14.950877 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.950886 4866 generic.go:356] "Generic (PLEG): container finished" podID="462bec67-536f-47f7-a649-860deadfa79d" containerID="f9fde80b5348eff8ac962ca8547e203c3eeec146e4ebe9bf2150b13d0288fafc" exitCode=0 Nov 02 13:57:14.950877 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.950892 4866 generic.go:356] "Generic (PLEG): container finished" podID="462bec67-536f-47f7-a649-860deadfa79d" containerID="67eab4a34be1ae5d3e95addfea868eec418679fd4f81fc2f99f110a3f08adf57" exitCode=0 Nov 02 13:57:14.951298 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.950897 4866 generic.go:356] "Generic (PLEG): container finished" podID="462bec67-536f-47f7-a649-860deadfa79d" containerID="42e327c8ce1d010e2ab541fe24b31f91ac981fe8a1a3a72c160d6a5ba7e8bffc" exitCode=0 Nov 02 13:57:14.966925 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.966896 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/1.log" Nov 02 13:57:14.966925 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.966927 4866 generic.go:356] "Generic (PLEG): container finished" podID="b97f6b80-f53b-4fff-97f6-dff42fb5875f" containerID="f892c402188f1cfca2d6a95a0b536190c1ff017b4bf04c09813676621b9ec58d" exitCode=255 Nov 02 13:57:14.974160 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.974123 4866 generic.go:356] "Generic (PLEG): container finished" podID="baf43fff2e8eac9bddd0973d4b16dfc0" containerID="74117fd2c594078dbb559421671d1d1d99d5ae5782fc5858452d8ef2e768d56c" exitCode=0 Nov 02 13:57:14.981615 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.981590 4866 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 13:57:14.983979 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.983935 4866 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:57:14.984139 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.983979 4866 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:57:14.984139 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.983992 4866 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:57:14.997210 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.997184 4866 manager.go:324] Recovery completed Nov 02 13:57:14.998828 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:14.998807 4866 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:57:15.016918 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.016833 4866 cpu_manager.go:226] "Starting" policy="none" Nov 02 13:57:15.016918 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.016860 4866 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 13:57:15.016918 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.016889 4866 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 13:57:15.017190 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.017124 4866 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 13:57:15.017190 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.017142 4866 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 13:57:15.017190 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.017169 4866 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 13:57:15.017190 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.017179 4866 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 13:57:15.017190 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.017186 4866 policy_none.go:50] "Start" Nov 02 13:57:15.017427 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.017207 4866 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 13:57:15.017427 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.017222 4866 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 13:57:15.017514 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.017484 4866 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 13:57:15.017514 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.017497 4866 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 13:57:15.017514 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.017505 4866 policy_none.go:44] "Start" Nov 02 13:57:15.022744 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.022709 4866 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 13:57:15.028350 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.028331 4866 manager.go:341] "Starting Device Plugin manager" Nov 02 13:57:15.028797 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:15.028359 4866 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 13:57:15.028861 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.028800 4866 server.go:87] "Starting device plugin registration server" Nov 02 13:57:15.029089 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.029073 4866 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 13:57:15.029175 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.029087 4866 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 13:57:15.029231 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.029186 4866 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 13:57:15.029327 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.029298 4866 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 13:57:15.029399 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.029333 4866 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 13:57:15.029399 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.029340 4866 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 13:57:15.029481 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.029402 4866 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T13:57:15.029323134Z","UUID":"3dbe3842-cd30-42fd-9a26-ccfb1bd39ac9","Handler":null,"Name":"","Endpoint":""} Nov 02 13:57:15.030622 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:15.030601 4866 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 13:57:15.030707 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.030689 4866 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 13:57:15.030769 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.030759 4866 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 13:57:15.038166 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.038122 4866 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal"] Nov 02 13:57:15.038883 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.038866 4866 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" Nov 02 13:57:15.039063 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.039008 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" event={"ID":"baf43fff2e8eac9bddd0973d4b16dfc0","Type":"ContainerStarted","Data":"f1db62f7d8d6d2283ffd3ad0cdfbc65062b11c787bc44231f5d0b252149978b2"} Nov 02 13:57:15.039126 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.039085 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" event={"ID":"baf43fff2e8eac9bddd0973d4b16dfc0","Type":"ContainerDied","Data":"74117fd2c594078dbb559421671d1d1d99d5ae5782fc5858452d8ef2e768d56c"} Nov 02 13:57:15.039126 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.039096 4866 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" Nov 02 13:57:15.039126 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.039104 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" event={"ID":"baf43fff2e8eac9bddd0973d4b16dfc0","Type":"ContainerStarted","Data":"a50b491b0757cf499544e39ae2a906f296f884e34c2263e402a1659550dc730d"} Nov 02 13:57:15.039291 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.039128 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" event={"ID":"fcd82cc88301759f1e08e2b014a50651","Type":"ContainerStarted","Data":"939b74f88148ca707b694989b860a78a22deb89e81374a3b076152ba8054f3fd"} Nov 02 13:57:15.039291 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.039140 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" event={"ID":"fcd82cc88301759f1e08e2b014a50651","Type":"ContainerStarted","Data":"cb03afa4c51dff071823c8c1d7f6ff6f927247e4ee5f9d1a4132d0dcb227b90c"} Nov 02 13:57:15.089920 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.089880 4866 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 13:57:15.090093 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:15.089965 4866 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-99.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" Nov 02 13:57:15.090506 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.090490 4866 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 13:57:15.090539 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:15.090528 4866 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" Nov 02 13:57:15.129411 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.129367 4866 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:57:15.131120 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.131102 4866 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:57:15.131188 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.131134 4866 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:57:15.131188 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.131145 4866 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-99.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:57:15.131256 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.131189 4866 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-99.ec2.internal" Nov 02 13:57:15.182165 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.182122 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/baf43fff2e8eac9bddd0973d4b16dfc0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal\" (UID: \"baf43fff2e8eac9bddd0973d4b16dfc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" Nov 02 13:57:15.182296 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.182171 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/baf43fff2e8eac9bddd0973d4b16dfc0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal\" (UID: \"baf43fff2e8eac9bddd0973d4b16dfc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" Nov 02 13:57:15.182296 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.182190 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcd82cc88301759f1e08e2b014a50651-config\") pod \"kube-apiserver-proxy-ip-10-0-134-99.ec2.internal\" (UID: \"fcd82cc88301759f1e08e2b014a50651\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" Nov 02 13:57:15.196016 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.195988 4866 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-99.ec2.internal" Nov 02 13:57:15.196122 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.196070 4866 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-99.ec2.internal" Nov 02 13:57:15.283082 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.282987 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/baf43fff2e8eac9bddd0973d4b16dfc0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal\" (UID: \"baf43fff2e8eac9bddd0973d4b16dfc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" Nov 02 13:57:15.283082 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.283007 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/baf43fff2e8eac9bddd0973d4b16dfc0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal\" (UID: \"baf43fff2e8eac9bddd0973d4b16dfc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" Nov 02 13:57:15.283082 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.283050 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/baf43fff2e8eac9bddd0973d4b16dfc0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal\" (UID: \"baf43fff2e8eac9bddd0973d4b16dfc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" Nov 02 13:57:15.283082 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.283075 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcd82cc88301759f1e08e2b014a50651-config\") pod \"kube-apiserver-proxy-ip-10-0-134-99.ec2.internal\" (UID: \"fcd82cc88301759f1e08e2b014a50651\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" Nov 02 13:57:15.283308 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.283113 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcd82cc88301759f1e08e2b014a50651-config\") pod \"kube-apiserver-proxy-ip-10-0-134-99.ec2.internal\" (UID: \"fcd82cc88301759f1e08e2b014a50651\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" Nov 02 13:57:15.283308 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.283145 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/baf43fff2e8eac9bddd0973d4b16dfc0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal\" (UID: \"baf43fff2e8eac9bddd0973d4b16dfc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" Nov 02 13:57:15.888345 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.888308 4866 apiserver.go:52] "Watching apiserver" Nov 02 13:57:15.940644 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.940613 4866 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 13:57:15.941312 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.941284 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-frx4v","openshift-network-diagnostics/network-check-source-6f78d59f99-ljkcn","kube-system/global-pull-secret-syncer-kv25k","openshift-ingress/router-default-7899fcdf65-hl87t","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8","openshift-console-operator/console-operator-7cd56b8dbc-5hfrp","openshift-dns/dns-default-cp4vx","openshift-network-diagnostics/network-check-target-pstlx","kube-system/kas-connection-checker-685895c7c9-clfrl","openshift-insights/insights-operator-7c4986d449-fg7tg","openshift-ingress-canary/ingress-canary-f9kv5","openshift-multus/multus-8xfsb","openshift-multus/multus-additional-cni-plugins-ttcsk","openshift-image-registry/node-ca-2sxls","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal","openshift-image-registry/image-registry-b64d99df4-mtsvs","openshift-image-registry/image-registry-579f678cff-m54g4","openshift-insights/insights-runtime-extractor-x6kpv","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm","openshift-network-console/networking-console-plugin-86c478bb88-hvxsl","kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal","openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq","kube-system/kas-connection-checker-685895c7c9-mbpqq","openshift-dns/node-resolver-zgxlv","openshift-multus/network-metrics-daemon-r5vdt","openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5","openshift-cluster-node-tuning-operator/tuned-v47w5","kube-system/konnectivity-agent-5vmgp","openshift-ovn-kubernetes/ovnkube-node-jg9lw","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xrfvm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p","openshift-network-operator/iptables-alerter-sdcn4"] Nov 02 13:57:15.942454 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.942437 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:15.943351 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.943324 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:15.944207 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.944179 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:15.944508 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.944492 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:15.945634 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.945617 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:15.949293 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.949276 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:15.949772 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.949760 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:57:15.956028 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.956008 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:15.956530 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.956511 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:15.957260 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.957246 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 13:57:15.957434 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.957418 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 13:57:15.957482 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.957466 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 13:57:15.957518 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.957495 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 13:57:15.957577 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.957560 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 13:57:15.957633 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.957622 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 13:57:15.957678 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.957650 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 13:57:15.957724 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.957704 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:57:15.957908 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.957890 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 13:57:15.958029 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.957921 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mfgzk\"" Nov 02 13:57:15.958029 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.957985 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 13:57:15.958177 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.958163 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-lpst2\"" Nov 02 13:57:15.958645 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.958629 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 13:57:15.987934 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.987904 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 13:57:15.987934 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.987917 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 13:57:15.988153 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.987903 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 13:57:15.988153 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988013 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 13:57:15.988153 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988025 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 13:57:15.988153 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988038 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:57:15.988153 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988057 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 13:57:15.988153 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988063 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 13:57:15.988153 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988111 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 13:57:15.988153 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988141 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 13:57:15.988153 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988142 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988154 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988172 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988174 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988261 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988191 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988192 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988209 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988226 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988286 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988235 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988305 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988332 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988347 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988368 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988373 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988385 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988409 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988433 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988227 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988468 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988485 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988496 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988470 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988516 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988519 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988547 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988553 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988573 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988585 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988593 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988590 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 13:57:15.988593 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988615 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988630 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988641 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988645 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988656 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988649 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988669 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988676 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988698 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988722 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988772 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988823 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988830 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988835 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xjjmq\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988833 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mbsf9\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.988875 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989013 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jk68j\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989075 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989146 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nkhq6\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989162 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989194 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-f9rnl\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989210 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989231 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989240 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-qxxgw\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989272 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-cd6m4\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989282 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kftg9\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989330 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m5vzv\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989344 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989370 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wlrs4\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989386 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-w7snh\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989438 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bhk6z\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989468 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2zrsm\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989501 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-8ms52\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989529 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4pvjb\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989568 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-75928\"" Nov 02 13:57:15.989711 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989655 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t4wpm\"" Nov 02 13:57:15.990680 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989752 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b9rw8\"" Nov 02 13:57:15.990680 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989787 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 13:57:15.990680 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989826 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 13:57:15.990680 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989869 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9cgsx\"" Nov 02 13:57:15.990680 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989890 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4gjwh\"" Nov 02 13:57:15.990680 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989908 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7l5rk\"" Nov 02 13:57:15.990680 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989960 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xm2nq\"" Nov 02 13:57:15.990680 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.989989 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 13:57:15.990680 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.990103 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-sczcr\"" Nov 02 13:57:15.990680 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.990160 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7mgxc\"" Nov 02 13:57:15.990680 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.990205 4866 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" Nov 02 13:57:15.990680 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.990338 4866 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" Nov 02 13:57:15.999304 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.999293 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 13:57:15.999304 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.999298 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 13:57:15.999746 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:15.999732 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 13:57:16.004424 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.002193 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 13:57:16.004424 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.002205 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 13:57:16.006123 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.006108 4866 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 13:57:16.012031 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.012018 4866 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 13:57:16.012108 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.012059 4866 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-99.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" Nov 02 13:57:16.012869 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.012857 4866 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 13:57:16.012905 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.012885 4866 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-99.ec2.internal" Nov 02 13:57:16.087724 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.087692 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b97f6b80-f53b-4fff-97f6-dff42fb5875f-serving-cert\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:16.087724 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.087725 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.087911 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.087742 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-hostroot\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.087911 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.087797 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-hvxsl\" (UID: \"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:16.087911 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.087814 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw75f\" (UniqueName: \"kubernetes.io/projected/87b5876f-0012-4bdf-b037-25070684476c-kube-api-access-bw75f\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:16.087911 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.087862 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.088099 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088048 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghtzt\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-kube-api-access-ghtzt\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:16.088099 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088064 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b97f6b80-f53b-4fff-97f6-dff42fb5875f-serving-cert\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:16.088099 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088091 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:16.088239 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088132 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-bound-sa-token\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:16.088239 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088155 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:16.088239 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088165 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-hvxsl\" (UID: \"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:16.088239 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088171 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-lib-modules\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.088239 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088188 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-tuned\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.088239 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088229 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-data-volume\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:16.088501 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088248 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:16.088501 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088265 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fr7m\" (UniqueName: \"kubernetes.io/projected/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-kube-api-access-9fr7m\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:16.088501 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088282 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/462bec67-536f-47f7-a649-860deadfa79d-system-cni-dir\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:57:16.088501 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088298 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-run-openvswitch\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.088501 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088304 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-tuned\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.088501 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088314 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-ovn-node-metrics-cert\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.088501 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088311 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:16.088501 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088454 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-data-volume\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:16.088501 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088471 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-installation-pull-secrets\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:16.088859 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088506 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-registration-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.088859 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088533 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-ovn-node-metrics-cert\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.088859 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088532 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-run-multus-certs\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.088859 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088587 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/462bec67-536f-47f7-a649-860deadfa79d-cnibin\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:57:16.088859 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088614 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-tmp-dir\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:16.088859 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088637 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-sys\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.088859 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088661 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-etc-openvswitch\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.088859 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088700 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-tmp-dir\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:16.088859 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088701 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-log-socket\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.088859 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088744 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-cni-netd\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.088859 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088751 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-installation-pull-secrets\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:16.088859 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088784 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-ovnkube-config\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.088859 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088810 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lvl4\" (UniqueName: \"kubernetes.io/projected/4f856415-9f37-415c-8dd5-65fc1ef08abc-kube-api-access-7lvl4\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:16.089327 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088906 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-run-ovn\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.089327 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088942 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:16.089327 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.088983 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-var-lib-cni-multus\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.089327 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089009 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:16.089327 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089032 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-ovnkube-config\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.089327 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089032 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-sysconfig\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.089327 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089073 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-systemd-units\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.089327 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089092 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-run-systemd\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.089327 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089117 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-bound-sa-token\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:16.089327 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089147 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/087ad47b-bcc1-4332-9706-8f0ed148252d-hosts-file\") pod \"node-resolver-zgxlv\" (UID: \"087ad47b-bcc1-4332-9706-8f0ed148252d\") " pod="openshift-dns/node-resolver-zgxlv" Nov 02 13:57:16.089327 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089174 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67v9\" (UniqueName: \"kubernetes.io/projected/7120b45c-1c5b-4c7c-863f-1902f96c00f5-kube-api-access-p67v9\") pod \"ingress-canary-f9kv5\" (UID: \"7120b45c-1c5b-4c7c-863f-1902f96c00f5\") " pod="openshift-ingress-canary/ingress-canary-f9kv5" Nov 02 13:57:16.089327 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089241 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f856415-9f37-415c-8dd5-65fc1ef08abc-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:16.089327 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089263 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17f96927-431c-4743-8cda-eb1856feab7a-cni-binary-copy\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.089327 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089278 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-run-netns\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.089327 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089294 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/462bec67-536f-47f7-a649-860deadfa79d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:57:16.089327 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089314 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-kubernetes\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.089925 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089349 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tht9k\" (UniqueName: \"kubernetes.io/projected/b1259f5f-470e-4ca5-8760-37bff1735835-kube-api-access-tht9k\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:16.089925 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089401 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hvxsl\" (UID: \"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:16.089925 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089435 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-crio-socket\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:16.089925 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089502 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prcq\" (UniqueName: \"kubernetes.io/projected/a8133c4b-08bc-443b-aa8c-d914da38a2a5-kube-api-access-9prcq\") pod \"network-check-source-6f78d59f99-ljkcn\" (UID: \"a8133c4b-08bc-443b-aa8c-d914da38a2a5\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ljkcn" Nov 02 13:57:16.089925 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089507 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f856415-9f37-415c-8dd5-65fc1ef08abc-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:16.089925 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089543 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-multus-cni-dir\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.089925 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089579 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17f96927-431c-4743-8cda-eb1856feab7a-cni-binary-copy\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.089925 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089620 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/462bec67-536f-47f7-a649-860deadfa79d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:57:16.089925 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089646 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:16.089925 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089668 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f651d09a-35a5-4a64-8de3-b824dcfd90ef-iptables-alerter-script\") pod \"iptables-alerter-sdcn4\" (UID: \"f651d09a-35a5-4a64-8de3-b824dcfd90ef\") " pod="openshift-network-operator/iptables-alerter-sdcn4" Nov 02 13:57:16.089925 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089704 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-sysctl-conf\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.089925 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089751 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-env-overrides\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.089925 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089825 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:16.089925 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089851 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-etc-kubernetes\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.089925 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089860 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-env-overrides\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.089925 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089872 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-system-cni-dir\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.089925 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089908 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hbn9\" (UniqueName: \"kubernetes.io/projected/d445fa88-f52d-460f-b89a-61de9cc8d541-kube-api-access-9hbn9\") pod \"kas-connection-checker-685895c7c9-mbpqq\" (UID: \"d445fa88-f52d-460f-b89a-61de9cc8d541\") " pod="kube-system/kas-connection-checker-685895c7c9-mbpqq" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089941 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvxbp\" (UniqueName: \"kubernetes.io/projected/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-kube-api-access-wvxbp\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089983 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebd0d655-5116-4a35-89ff-f7575026d4ea-ca-trust-extracted\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.089989 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f651d09a-35a5-4a64-8de3-b824dcfd90ef-iptables-alerter-script\") pod \"iptables-alerter-sdcn4\" (UID: \"f651d09a-35a5-4a64-8de3-b824dcfd90ef\") " pod="openshift-network-operator/iptables-alerter-sdcn4" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090008 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-installation-pull-secrets\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090095 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebd0d655-5116-4a35-89ff-f7575026d4ea-ca-trust-extracted\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090112 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-systemd\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090152 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-tmp\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090169 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4f856415-9f37-415c-8dd5-65fc1ef08abc-snapshots\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090185 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-ovnkube-script-lib\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090189 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-installation-pull-secrets\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090199 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b97f6b80-f53b-4fff-97f6-dff42fb5875f-config\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090217 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5p6j\" (UniqueName: \"kubernetes.io/projected/b97f6b80-f53b-4fff-97f6-dff42fb5875f-kube-api-access-t5p6j\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090236 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m66q\" (UniqueName: \"kubernetes.io/projected/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-api-access-6m66q\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090244 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-tmp\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090262 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc0287f6-d102-43a4-80cd-62b3acfb3991-config\") pod \"service-ca-operator-544d749c8c-zv6s5\" (UID: \"fc0287f6-d102-43a4-80cd-62b3acfb3991\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090270 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4f856415-9f37-415c-8dd5-65fc1ef08abc-snapshots\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090289 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-device-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090314 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-var-lib-kubelet\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090344 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-cni-bin\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090369 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7120b45c-1c5b-4c7c-863f-1902f96c00f5-cert\") pod \"ingress-canary-f9kv5\" (UID: \"7120b45c-1c5b-4c7c-863f-1902f96c00f5\") " pod="openshift-ingress-canary/ingress-canary-f9kv5" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090402 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-run-k8s-cni-cncf-io\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090420 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-ovnkube-script-lib\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090425 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b97f6b80-f53b-4fff-97f6-dff42fb5875f-config\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090443 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e9b81de7-b765-4eac-afd2-86d99495af3d-serviceca\") pod \"node-ca-2sxls\" (UID: \"e9b81de7-b765-4eac-afd2-86d99495af3d\") " pod="openshift-image-registry/node-ca-2sxls" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090462 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc0287f6-d102-43a4-80cd-62b3acfb3991-config\") pod \"service-ca-operator-544d749c8c-zv6s5\" (UID: \"fc0287f6-d102-43a4-80cd-62b3acfb3991\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090473 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-socket-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090551 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7120b45c-1c5b-4c7c-863f-1902f96c00f5-cert\") pod \"ingress-canary-f9kv5\" (UID: \"7120b45c-1c5b-4c7c-863f-1902f96c00f5\") " pod="openshift-ingress-canary/ingress-canary-f9kv5" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090588 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgqq8\" (UniqueName: \"kubernetes.io/projected/f57dae1b-545a-42d4-92f8-19c79e9de22d-kube-api-access-qgqq8\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090618 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trjsq\" (UniqueName: \"kubernetes.io/projected/a7095b0b-813f-4b60-989f-8a01e2743bbe-kube-api-access-trjsq\") pod \"kas-connection-checker-685895c7c9-frx4v\" (UID: \"a7095b0b-813f-4b60-989f-8a01e2743bbe\") " pod="kube-system/kas-connection-checker-685895c7c9-frx4v" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090631 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e9b81de7-b765-4eac-afd2-86d99495af3d-serviceca\") pod \"node-ca-2sxls\" (UID: \"e9b81de7-b765-4eac-afd2-86d99495af3d\") " pod="openshift-image-registry/node-ca-2sxls" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090648 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tss59\" (UniqueName: \"kubernetes.io/projected/cf94ad7e-96a0-4d35-9cb3-ad83f8234281-kube-api-access-tss59\") pod \"volume-data-source-validator-58c9f5f46d-xrfvm\" (UID: \"cf94ad7e-96a0-4d35-9cb3-ad83f8234281\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xrfvm" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090675 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd3bd8d9-b081-48bd-be29-582afe6f82fd-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm\" (UID: \"bd3bd8d9-b081-48bd-be29-582afe6f82fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" Nov 02 13:57:16.090699 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090700 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7ms9\" (UniqueName: \"kubernetes.io/projected/bd3bd8d9-b081-48bd-be29-582afe6f82fd-kube-api-access-d7ms9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm\" (UID: \"bd3bd8d9-b081-48bd-be29-582afe6f82fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090749 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-image-registry-private-configuration\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090777 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-certificates\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090802 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-default-certificate\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090829 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/462bec67-536f-47f7-a649-860deadfa79d-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090857 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd3bd8d9-b081-48bd-be29-582afe6f82fd-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm\" (UID: \"bd3bd8d9-b081-48bd-be29-582afe6f82fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090859 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f856415-9f37-415c-8dd5-65fc1ef08abc-service-ca-bundle\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090911 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-image-registry-private-configuration\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090941 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f212522d-ef61-4b05-aee8-538433003f78-ca-trust-extracted\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090985 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-registry-certificates\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.090992 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-certificates\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091011 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75khg\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-kube-api-access-75khg\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091016 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f856415-9f37-415c-8dd5-65fc1ef08abc-service-ca-bundle\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091022 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-image-registry-private-configuration\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091025 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-default-certificate\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091042 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-stats-auth\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091068 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f212522d-ef61-4b05-aee8-538433003f78-ca-trust-extracted\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091073 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1e29491-8fb3-4918-b930-796653f412ec-dbus\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091043 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/462bec67-536f-47f7-a649-860deadfa79d-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091125 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/462bec67-536f-47f7-a649-860deadfa79d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091145 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-image-registry-private-configuration\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091152 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9b81de7-b765-4eac-afd2-86d99495af3d-host\") pod \"node-ca-2sxls\" (UID: \"e9b81de7-b765-4eac-afd2-86d99495af3d\") " pod="openshift-image-registry/node-ca-2sxls" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091161 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-registry-certificates\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091174 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-host\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091201 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-run-netns\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091217 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-stats-auth\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091236 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc0287f6-d102-43a4-80cd-62b3acfb3991-serving-cert\") pod \"service-ca-operator-544d749c8c-zv6s5\" (UID: \"fc0287f6-d102-43a4-80cd-62b3acfb3991\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091316 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-cnibin\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091352 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc0287f6-d102-43a4-80cd-62b3acfb3991-serving-cert\") pod \"service-ca-operator-544d749c8c-zv6s5\" (UID: \"fc0287f6-d102-43a4-80cd-62b3acfb3991\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091360 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-slash\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091389 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xctln\" (UniqueName: \"kubernetes.io/projected/fc0287f6-d102-43a4-80cd-62b3acfb3991-kube-api-access-xctln\") pod \"service-ca-operator-544d749c8c-zv6s5\" (UID: \"fc0287f6-d102-43a4-80cd-62b3acfb3991\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091405 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bd3bd8d9-b081-48bd-be29-582afe6f82fd-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm\" (UID: \"bd3bd8d9-b081-48bd-be29-582afe6f82fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091445 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmrvz\" (UniqueName: \"kubernetes.io/projected/f651d09a-35a5-4a64-8de3-b824dcfd90ef-kube-api-access-vmrvz\") pod \"iptables-alerter-sdcn4\" (UID: \"f651d09a-35a5-4a64-8de3-b824dcfd90ef\") " pod="openshift-network-operator/iptables-alerter-sdcn4" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091473 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091501 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091526 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-var-lib-kubelet\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091560 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q67r\" (UniqueName: \"kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r\") pod \"network-check-target-pstlx\" (UID: \"7f34d424-89e7-4881-bd4f-8e77566169ed\") " pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091595 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091623 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-trusted-ca\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091639 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bd3bd8d9-b081-48bd-be29-582afe6f82fd-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm\" (UID: \"bd3bd8d9-b081-48bd-be29-582afe6f82fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091654 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/462bec67-536f-47f7-a649-860deadfa79d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091749 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-modprobe-d\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091787 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-run\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091814 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/087ad47b-bcc1-4332-9706-8f0ed148252d-tmp-dir\") pod \"node-resolver-zgxlv\" (UID: \"087ad47b-bcc1-4332-9706-8f0ed148252d\") " pod="openshift-dns/node-resolver-zgxlv" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091868 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72rpv\" (UniqueName: \"kubernetes.io/projected/087ad47b-bcc1-4332-9706-8f0ed148252d-kube-api-access-72rpv\") pod \"node-resolver-zgxlv\" (UID: \"087ad47b-bcc1-4332-9706-8f0ed148252d\") " pod="openshift-dns/node-resolver-zgxlv" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091876 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/462bec67-536f-47f7-a649-860deadfa79d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091897 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/087ad47b-bcc1-4332-9706-8f0ed148252d-tmp-dir\") pod \"node-resolver-zgxlv\" (UID: \"087ad47b-bcc1-4332-9706-8f0ed148252d\") " pod="openshift-dns/node-resolver-zgxlv" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091905 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-trusted-ca\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091935 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4zbv\" (UniqueName: \"kubernetes.io/projected/7157562b-09b1-44a1-95f8-f29c7e62e475-kube-api-access-z4zbv\") pod \"kas-connection-checker-685895c7c9-clfrl\" (UID: \"7157562b-09b1-44a1-95f8-f29c7e62e475\") " pod="kube-system/kas-connection-checker-685895c7c9-clfrl" Nov 02 13:57:16.091965 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.091985 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092010 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1e29491-8fb3-4918-b930-796653f412ec-kubelet-config\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092037 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a722d-ff18-4837-8169-f1a642a268bb-konnectivity-ca\") pod \"konnectivity-agent-5vmgp\" (UID: \"aa6a722d-ff18-4837-8169-f1a642a268bb\") " pod="kube-system/konnectivity-agent-5vmgp" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092141 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s78vb\" (UniqueName: \"kubernetes.io/projected/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-kube-api-access-s78vb\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092162 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b1259f5f-470e-4ca5-8760-37bff1735835-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092182 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092201 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092240 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqtvw\" (UniqueName: \"kubernetes.io/projected/462bec67-536f-47f7-a649-860deadfa79d-kube-api-access-mqtvw\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092306 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a722d-ff18-4837-8169-f1a642a268bb-konnectivity-ca\") pod \"konnectivity-agent-5vmgp\" (UID: \"aa6a722d-ff18-4837-8169-f1a642a268bb\") " pod="kube-system/konnectivity-agent-5vmgp" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092375 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-config-volume\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092373 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1e29491-8fb3-4918-b930-796653f412ec-original-pull-secret\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092435 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa6a722d-ff18-4837-8169-f1a642a268bb-agent-certs\") pod \"konnectivity-agent-5vmgp\" (UID: \"aa6a722d-ff18-4837-8169-f1a642a268bb\") " pod="kube-system/konnectivity-agent-5vmgp" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092456 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-kubelet\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092484 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-trusted-ca\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092493 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b1259f5f-470e-4ca5-8760-37bff1735835-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092510 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr8lv\" (UniqueName: \"kubernetes.io/projected/17f96927-431c-4743-8cda-eb1856feab7a-kube-api-access-mr8lv\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092535 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f856415-9f37-415c-8dd5-65fc1ef08abc-serving-cert\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092539 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-config-volume\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092562 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-sysctl-d\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092589 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f856415-9f37-415c-8dd5-65fc1ef08abc-tmp\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092615 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-var-lib-cni-bin\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092656 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/17f96927-431c-4743-8cda-eb1856feab7a-multus-daemon-config\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092660 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f856415-9f37-415c-8dd5-65fc1ef08abc-serving-cert\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092687 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b97f6b80-f53b-4fff-97f6-dff42fb5875f-trusted-ca\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092698 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f856415-9f37-415c-8dd5-65fc1ef08abc-tmp\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092698 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa6a722d-ff18-4837-8169-f1a642a268bb-agent-certs\") pod \"konnectivity-agent-5vmgp\" (UID: \"aa6a722d-ff18-4837-8169-f1a642a268bb\") " pod="kube-system/konnectivity-agent-5vmgp" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092794 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-multus-socket-dir-parent\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092804 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-trusted-ca\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092837 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/17f96927-431c-4743-8cda-eb1856feab7a-multus-daemon-config\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092874 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bwcd\" (UniqueName: \"kubernetes.io/projected/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-kube-api-access-8bwcd\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092912 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6b9z\" (UniqueName: \"kubernetes.io/projected/e9b81de7-b765-4eac-afd2-86d99495af3d-kube-api-access-g6b9z\") pod \"node-ca-2sxls\" (UID: \"e9b81de7-b765-4eac-afd2-86d99495af3d\") " pod="openshift-image-registry/node-ca-2sxls" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.092987 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-var-lib-openvswitch\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.093006 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-run-ovn-kubernetes\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.093023 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-sys-fs\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.093041 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-multus-conf-dir\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.093051 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b97f6b80-f53b-4fff-97f6-dff42fb5875f-trusted-ca\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.093087 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f651d09a-35a5-4a64-8de3-b824dcfd90ef-host-slash\") pod \"iptables-alerter-sdcn4\" (UID: \"f651d09a-35a5-4a64-8de3-b824dcfd90ef\") " pod="openshift-network-operator/iptables-alerter-sdcn4" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.093128 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvxgk\" (UniqueName: \"kubernetes.io/projected/4441e34f-75c1-40d0-9d06-af34fd3fbb50-kube-api-access-vvxgk\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:57:16.093616 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.093245 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-node-log\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.113685 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.113648 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lvl4\" (UniqueName: \"kubernetes.io/projected/4f856415-9f37-415c-8dd5-65fc1ef08abc-kube-api-access-7lvl4\") pod \"insights-operator-7c4986d449-fg7tg\" (UID: \"4f856415-9f37-415c-8dd5-65fc1ef08abc\") " pod="openshift-insights/insights-operator-7c4986d449-fg7tg" Nov 02 13:57:16.113811 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.113650 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tht9k\" (UniqueName: \"kubernetes.io/projected/b1259f5f-470e-4ca5-8760-37bff1735835-kube-api-access-tht9k\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:16.118751 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.118730 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75khg\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-kube-api-access-75khg\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:16.119827 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.119803 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xctln\" (UniqueName: \"kubernetes.io/projected/fc0287f6-d102-43a4-80cd-62b3acfb3991-kube-api-access-xctln\") pod \"service-ca-operator-544d749c8c-zv6s5\" (UID: \"fc0287f6-d102-43a4-80cd-62b3acfb3991\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" Nov 02 13:57:16.119913 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.119826 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bwcd\" (UniqueName: \"kubernetes.io/projected/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-kube-api-access-8bwcd\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:16.120922 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.120904 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-bound-sa-token\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:16.121268 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.121249 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7ms9\" (UniqueName: \"kubernetes.io/projected/bd3bd8d9-b081-48bd-be29-582afe6f82fd-kube-api-access-d7ms9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm\" (UID: \"bd3bd8d9-b081-48bd-be29-582afe6f82fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" Nov 02 13:57:16.121445 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.121425 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72rpv\" (UniqueName: \"kubernetes.io/projected/087ad47b-bcc1-4332-9706-8f0ed148252d-kube-api-access-72rpv\") pod \"node-resolver-zgxlv\" (UID: \"087ad47b-bcc1-4332-9706-8f0ed148252d\") " pod="openshift-dns/node-resolver-zgxlv" Nov 02 13:57:16.121488 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.121427 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tss59\" (UniqueName: \"kubernetes.io/projected/cf94ad7e-96a0-4d35-9cb3-ad83f8234281-kube-api-access-tss59\") pod \"volume-data-source-validator-58c9f5f46d-xrfvm\" (UID: \"cf94ad7e-96a0-4d35-9cb3-ad83f8234281\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xrfvm" Nov 02 13:57:16.123037 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.123017 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgqq8\" (UniqueName: \"kubernetes.io/projected/f57dae1b-545a-42d4-92f8-19c79e9de22d-kube-api-access-qgqq8\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.123122 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.123108 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghtzt\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-kube-api-access-ghtzt\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:16.123757 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.123728 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p67v9\" (UniqueName: \"kubernetes.io/projected/7120b45c-1c5b-4c7c-863f-1902f96c00f5-kube-api-access-p67v9\") pod \"ingress-canary-f9kv5\" (UID: \"7120b45c-1c5b-4c7c-863f-1902f96c00f5\") " pod="openshift-ingress-canary/ingress-canary-f9kv5" Nov 02 13:57:16.123933 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.123915 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvxbp\" (UniqueName: \"kubernetes.io/projected/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-kube-api-access-wvxbp\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.124025 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.124001 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hbn9\" (UniqueName: \"kubernetes.io/projected/d445fa88-f52d-460f-b89a-61de9cc8d541-kube-api-access-9hbn9\") pod \"kas-connection-checker-685895c7c9-mbpqq\" (UID: \"d445fa88-f52d-460f-b89a-61de9cc8d541\") " pod="kube-system/kas-connection-checker-685895c7c9-mbpqq" Nov 02 13:57:16.124307 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.124287 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-bound-sa-token\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:16.124437 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.124417 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmrvz\" (UniqueName: \"kubernetes.io/projected/f651d09a-35a5-4a64-8de3-b824dcfd90ef-kube-api-access-vmrvz\") pod \"iptables-alerter-sdcn4\" (UID: \"f651d09a-35a5-4a64-8de3-b824dcfd90ef\") " pod="openshift-network-operator/iptables-alerter-sdcn4" Nov 02 13:57:16.124618 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.124596 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q67r\" (UniqueName: \"kubernetes.io/projected/7f34d424-89e7-4881-bd4f-8e77566169ed-kube-api-access-7q67r\") pod \"network-check-target-pstlx\" (UID: \"7f34d424-89e7-4881-bd4f-8e77566169ed\") " pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:57:16.124913 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.124892 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4zbv\" (UniqueName: \"kubernetes.io/projected/7157562b-09b1-44a1-95f8-f29c7e62e475-kube-api-access-z4zbv\") pod \"kas-connection-checker-685895c7c9-clfrl\" (UID: \"7157562b-09b1-44a1-95f8-f29c7e62e475\") " pod="kube-system/kas-connection-checker-685895c7c9-clfrl" Nov 02 13:57:16.125042 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.124929 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6b9z\" (UniqueName: \"kubernetes.io/projected/e9b81de7-b765-4eac-afd2-86d99495af3d-kube-api-access-g6b9z\") pod \"node-ca-2sxls\" (UID: \"e9b81de7-b765-4eac-afd2-86d99495af3d\") " pod="openshift-image-registry/node-ca-2sxls" Nov 02 13:57:16.125150 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.125132 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9prcq\" (UniqueName: \"kubernetes.io/projected/a8133c4b-08bc-443b-aa8c-d914da38a2a5-kube-api-access-9prcq\") pod \"network-check-source-6f78d59f99-ljkcn\" (UID: \"a8133c4b-08bc-443b-aa8c-d914da38a2a5\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ljkcn" Nov 02 13:57:16.125291 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.125276 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5p6j\" (UniqueName: \"kubernetes.io/projected/b97f6b80-f53b-4fff-97f6-dff42fb5875f-kube-api-access-t5p6j\") pod \"console-operator-7cd56b8dbc-5hfrp\" (UID: \"b97f6b80-f53b-4fff-97f6-dff42fb5875f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:16.126355 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.126341 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trjsq\" (UniqueName: \"kubernetes.io/projected/a7095b0b-813f-4b60-989f-8a01e2743bbe-kube-api-access-trjsq\") pod \"kas-connection-checker-685895c7c9-frx4v\" (UID: \"a7095b0b-813f-4b60-989f-8a01e2743bbe\") " pod="kube-system/kas-connection-checker-685895c7c9-frx4v" Nov 02 13:57:16.128049 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.128033 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw75f\" (UniqueName: \"kubernetes.io/projected/87b5876f-0012-4bdf-b037-25070684476c-kube-api-access-bw75f\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:16.129153 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.129138 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvxgk\" (UniqueName: \"kubernetes.io/projected/4441e34f-75c1-40d0-9d06-af34fd3fbb50-kube-api-access-vvxgk\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:57:16.130418 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.130399 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqtvw\" (UniqueName: \"kubernetes.io/projected/462bec67-536f-47f7-a649-860deadfa79d-kube-api-access-mqtvw\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:57:16.131462 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.131442 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s78vb\" (UniqueName: \"kubernetes.io/projected/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-kube-api-access-s78vb\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.131557 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.131484 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr8lv\" (UniqueName: \"kubernetes.io/projected/17f96927-431c-4743-8cda-eb1856feab7a-kube-api-access-mr8lv\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.131557 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.131520 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m66q\" (UniqueName: \"kubernetes.io/projected/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-api-access-6m66q\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:16.131891 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.131858 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fr7m\" (UniqueName: \"kubernetes.io/projected/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-kube-api-access-9fr7m\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:16.193887 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.193861 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-sysctl-d\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.194058 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.193893 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-var-lib-cni-bin\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.194058 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.193909 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-multus-socket-dir-parent\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.194058 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.193925 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-var-lib-openvswitch\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.194058 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.193940 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-run-ovn-kubernetes\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.194058 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194002 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-run-ovn-kubernetes\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.194058 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194004 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-var-lib-cni-bin\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.194058 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194017 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-multus-socket-dir-parent\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.194058 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194022 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-var-lib-openvswitch\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.194058 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194050 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-sys-fs\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194072 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-sysctl-d\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194074 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-multus-conf-dir\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194095 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-multus-conf-dir\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194101 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f651d09a-35a5-4a64-8de3-b824dcfd90ef-host-slash\") pod \"iptables-alerter-sdcn4\" (UID: \"f651d09a-35a5-4a64-8de3-b824dcfd90ef\") " pod="openshift-network-operator/iptables-alerter-sdcn4" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194113 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-sys-fs\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194124 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-node-log\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194137 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f651d09a-35a5-4a64-8de3-b824dcfd90ef-host-slash\") pod \"iptables-alerter-sdcn4\" (UID: \"f651d09a-35a5-4a64-8de3-b824dcfd90ef\") " pod="openshift-network-operator/iptables-alerter-sdcn4" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194148 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194163 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-hostroot\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194181 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194200 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194195 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-node-log\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194226 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-hostroot\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194234 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194238 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194294 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-lib-modules\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.194305 4866 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194319 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194338 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/462bec67-536f-47f7-a649-860deadfa79d-system-cni-dir\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194352 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-run-openvswitch\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.194386 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls podName:ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:16.694364103 +0000 UTC m=+1.878211435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls") pod "dns-default-cp4vx" (UID: "ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e") : secret "dns-default-metrics-tls" not found Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194393 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-run-openvswitch\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.194403 4866 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194406 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-registration-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194435 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-run-multus-certs\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.194453 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-rbac-proxy-cm podName:d32c74f9-dc27-4e9f-a20b-da4c3704acc1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:16.694439388 +0000 UTC m=+1.878286754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x6kpv" (UID: "d32c74f9-dc27-4e9f-a20b-da4c3704acc1") : configmap "kube-rbac-proxy" not found Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194456 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-lib-modules\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194472 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-run-multus-certs\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194477 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/462bec67-536f-47f7-a649-860deadfa79d-cnibin\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:57:16.194474 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194400 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/462bec67-536f-47f7-a649-860deadfa79d-system-cni-dir\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194501 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-registration-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194502 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-sys\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194523 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/462bec67-536f-47f7-a649-860deadfa79d-cnibin\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194549 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-sys\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194559 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-etc-openvswitch\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194585 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-log-socket\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194611 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-log-socket\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194625 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-etc-openvswitch\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194630 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-cni-netd\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194656 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-run-ovn\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194657 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-cni-netd\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194672 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194686 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-run-ovn\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194695 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-var-lib-cni-multus\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194711 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194728 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-sysconfig\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194738 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-var-lib-cni-multus\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194741 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-systemd-units\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194771 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-systemd-units\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194773 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-run-systemd\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.194771 4866 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.194794 4866 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579f678cff-m54g4: secret "image-registry-tls" not found Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.194824 4866 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194828 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-sysconfig\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.194853 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls podName:f212522d-ef61-4b05-aee8-538433003f78 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:16.694840082 +0000 UTC m=+1.878687405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls") pod "image-registry-579f678cff-m54g4" (UID: "f212522d-ef61-4b05-aee8-538433003f78") : secret "image-registry-tls" not found Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194867 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-run-systemd\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.194872 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs podName:87b5876f-0012-4bdf-b037-25070684476c nodeName:}" failed. No retries permitted until 2026-11-02 13:57:16.69486342 +0000 UTC m=+1.878710747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs") pod "router-default-7899fcdf65-hl87t" (UID: "87b5876f-0012-4bdf-b037-25070684476c") : secret "router-metrics-certs-default" not found Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194896 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/087ad47b-bcc1-4332-9706-8f0ed148252d-hosts-file\") pod \"node-resolver-zgxlv\" (UID: \"087ad47b-bcc1-4332-9706-8f0ed148252d\") " pod="openshift-dns/node-resolver-zgxlv" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194916 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-run-netns\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194931 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-kubernetes\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194966 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hvxsl\" (UID: \"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194933 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/087ad47b-bcc1-4332-9706-8f0ed148252d-hosts-file\") pod \"node-resolver-zgxlv\" (UID: \"087ad47b-bcc1-4332-9706-8f0ed148252d\") " pod="openshift-dns/node-resolver-zgxlv" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.194985 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-crio-socket\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195006 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-multus-cni-dir\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195008 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-run-netns\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195024 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195031 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-kubernetes\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.195040 4866 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195041 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-sysctl-conf\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195056 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-crio-socket\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195074 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195089 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-multus-cni-dir\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.195098 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert podName:cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:16.695083704 +0000 UTC m=+1.878931034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hvxsl" (UID: "cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1") : secret "networking-console-plugin-cert" not found Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.195124 4866 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.195127 4866 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195131 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-etc-kubernetes\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.195161 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls podName:d32c74f9-dc27-4e9f-a20b-da4c3704acc1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:16.695147595 +0000 UTC m=+1.878994932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x6kpv" (UID: "d32c74f9-dc27-4e9f-a20b-da4c3704acc1") : secret "insights-runtime-extractor-tls" not found Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195168 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-sysctl-conf\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.195179 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls podName:b1259f5f-470e-4ca5-8760-37bff1735835 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:16.69516889 +0000 UTC m=+1.879016208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4tgbq" (UID: "b1259f5f-470e-4ca5-8760-37bff1735835") : secret "cluster-monitoring-operator-tls" not found Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195177 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-etc-kubernetes\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195196 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-system-cni-dir\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195219 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-systemd\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195238 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-device-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195252 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-var-lib-kubelet\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195267 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-cni-bin\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195280 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-systemd\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195284 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-run-k8s-cni-cncf-io\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195305 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-device-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195310 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-socket-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195319 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-var-lib-kubelet\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195320 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-system-cni-dir\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195333 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-cni-bin\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195346 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-host-run-k8s-cni-cncf-io\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195357 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1e29491-8fb3-4918-b930-796653f412ec-dbus\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195361 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-socket-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195375 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/462bec67-536f-47f7-a649-860deadfa79d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195390 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9b81de7-b765-4eac-afd2-86d99495af3d-host\") pod \"node-ca-2sxls\" (UID: \"e9b81de7-b765-4eac-afd2-86d99495af3d\") " pod="openshift-image-registry/node-ca-2sxls" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195410 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-host\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195424 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1e29491-8fb3-4918-b930-796653f412ec-dbus\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195434 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-run-netns\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195445 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/462bec67-536f-47f7-a649-860deadfa79d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttcsk\" (UID: \"462bec67-536f-47f7-a649-860deadfa79d\") " pod="openshift-multus/multus-additional-cni-plugins-ttcsk" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195455 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-host\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195459 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-cnibin\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195469 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-run-netns\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195481 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-slash\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195458 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9b81de7-b765-4eac-afd2-86d99495af3d-host\") pod \"node-ca-2sxls\" (UID: \"e9b81de7-b765-4eac-afd2-86d99495af3d\") " pod="openshift-image-registry/node-ca-2sxls" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195502 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195501 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17f96927-431c-4743-8cda-eb1856feab7a-cnibin\") pod \"multus-8xfsb\" (UID: \"17f96927-431c-4743-8cda-eb1856feab7a\") " pod="openshift-multus/multus-8xfsb" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195525 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195546 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-slash\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.195561 4866 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.195570 4866 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b64d99df4-mtsvs: secret "image-registry-tls" not found Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.195578 4866 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 13:57:16.195539 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.195603 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls podName:ebd0d655-5116-4a35-89ff-f7575026d4ea nodeName:}" failed. No retries permitted until 2026-11-02 13:57:16.695592968 +0000 UTC m=+1.879440302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls") pod "image-registry-b64d99df4-mtsvs" (UID: "ebd0d655-5116-4a35-89ff-f7575026d4ea") : secret "image-registry-tls" not found Nov 02 13:57:16.197903 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.195615 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs podName:4441e34f-75c1-40d0-9d06-af34fd3fbb50 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:16.695609785 +0000 UTC m=+1.879457100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs") pod "network-metrics-daemon-r5vdt" (UID: "4441e34f-75c1-40d0-9d06-af34fd3fbb50") : secret "metrics-daemon-secret" not found Nov 02 13:57:16.197903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195613 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-var-lib-kubelet\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.197903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195576 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-var-lib-kubelet\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.197903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195650 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.197903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195677 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-modprobe-d\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.197903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195700 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-run\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.197903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195710 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f57dae1b-545a-42d4-92f8-19c79e9de22d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zb5p\" (UID: \"f57dae1b-545a-42d4-92f8-19c79e9de22d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zb5p" Nov 02 13:57:16.197903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195775 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-etc-modprobe-d\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.197903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195772 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a-run\") pod \"tuned-v47w5\" (UID: \"b1fb0fb2-293e-43ea-8cae-e2c1235b8d9a\") " pod="openshift-cluster-node-tuning-operator/tuned-v47w5" Nov 02 13:57:16.197903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195805 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:16.197903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195832 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1e29491-8fb3-4918-b930-796653f412ec-kubelet-config\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:57:16.197903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195860 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:16.197903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195891 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-kubelet\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.197903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195894 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1e29491-8fb3-4918-b930-796653f412ec-kubelet-config\") pod \"global-pull-secret-syncer-kv25k\" (UID: \"d1e29491-8fb3-4918-b930-796653f412ec\") " pod="kube-system/global-pull-secret-syncer-kv25k" Nov 02 13:57:16.197903 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.195904 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle podName:87b5876f-0012-4bdf-b037-25070684476c nodeName:}" failed. No retries permitted until 2026-11-02 13:57:16.69589718 +0000 UTC m=+1.879744532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle") pod "router-default-7899fcdf65-hl87t" (UID: "87b5876f-0012-4bdf-b037-25070684476c") : configmap references non-existent config key: service-ca.crt Nov 02 13:57:16.197903 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.195924 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b39a3cfd-860d-48e5-bad7-68e4f2b4157a-host-kubelet\") pod \"ovnkube-node-jg9lw\" (UID: \"b39a3cfd-860d-48e5-bad7-68e4f2b4157a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:16.197903 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.195981 4866 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 13:57:16.197903 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.196011 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls podName:1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d nodeName:}" failed. No retries permitted until 2026-11-02 13:57:16.696001874 +0000 UTC m=+1.879849227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4sfb8" (UID: "1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d") : secret "samples-operator-tls" not found Nov 02 13:57:16.244434 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.244409 4866 scope.go:122] "RemoveContainer" containerID="f892c402188f1cfca2d6a95a0b536190c1ff017b4bf04c09813676621b9ec58d" Nov 02 13:57:16.699799 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.699761 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:16.699799 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.699805 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:16.700038 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.699831 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:16.700038 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.699851 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:16.700038 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.699888 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hvxsl\" (UID: \"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:16.700038 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.699918 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:16.700038 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.699920 4866 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 13:57:16.700038 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.699940 4866 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 13:57:16.700038 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.699978 4866 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579f678cff-m54g4: secret "image-registry-tls" not found Nov 02 13:57:16.700038 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.699964 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:16.700038 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.700018 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls podName:ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:17.699995991 +0000 UTC m=+2.883843330 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls") pod "dns-default-cp4vx" (UID: "ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e") : secret "dns-default-metrics-tls" not found Nov 02 13:57:16.700038 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.700038 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls podName:f212522d-ef61-4b05-aee8-538433003f78 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:17.700030881 +0000 UTC m=+2.883878196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls") pod "image-registry-579f678cff-m54g4" (UID: "f212522d-ef61-4b05-aee8-538433003f78") : secret "image-registry-tls" not found Nov 02 13:57:16.700509 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.700318 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:16.700670 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.700647 4866 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 13:57:16.700769 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.700679 4866 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 13:57:16.700769 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.700700 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls podName:b1259f5f-470e-4ca5-8760-37bff1735835 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:17.700688262 +0000 UTC m=+2.884535582 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4tgbq" (UID: "b1259f5f-470e-4ca5-8760-37bff1735835") : secret "cluster-monitoring-operator-tls" not found Nov 02 13:57:16.700769 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.700729 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs podName:87b5876f-0012-4bdf-b037-25070684476c nodeName:}" failed. No retries permitted until 2026-11-02 13:57:17.700718105 +0000 UTC m=+2.884565421 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs") pod "router-default-7899fcdf65-hl87t" (UID: "87b5876f-0012-4bdf-b037-25070684476c") : secret "router-metrics-certs-default" not found Nov 02 13:57:16.700873 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.700772 4866 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 13:57:16.700873 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.700815 4866 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 13:57:16.700873 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.700827 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls podName:d32c74f9-dc27-4e9f-a20b-da4c3704acc1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:17.700814044 +0000 UTC m=+2.884661374 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x6kpv" (UID: "d32c74f9-dc27-4e9f-a20b-da4c3704acc1") : secret "insights-runtime-extractor-tls" not found Nov 02 13:57:16.700873 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.700849 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert podName:cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:17.700839661 +0000 UTC m=+2.884687011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hvxsl" (UID: "cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1") : secret "networking-console-plugin-cert" not found Nov 02 13:57:16.701011 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.700869 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:16.701011 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.700893 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:57:16.701011 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.700924 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:16.701011 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.700973 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:16.701120 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.700981 4866 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 13:57:16.701120 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.701046 4866 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b64d99df4-mtsvs: secret "image-registry-tls" not found Nov 02 13:57:16.701120 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.701048 4866 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 13:57:16.701120 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.701033 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle podName:87b5876f-0012-4bdf-b037-25070684476c nodeName:}" failed. No retries permitted until 2026-11-02 13:57:17.70102468 +0000 UTC m=+2.884872013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle") pod "router-default-7899fcdf65-hl87t" (UID: "87b5876f-0012-4bdf-b037-25070684476c") : configmap references non-existent config key: service-ca.crt Nov 02 13:57:16.701120 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.700994 4866 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 13:57:16.701120 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.701103 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls podName:ebd0d655-5116-4a35-89ff-f7575026d4ea nodeName:}" failed. No retries permitted until 2026-11-02 13:57:17.701089772 +0000 UTC m=+2.884937095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls") pod "image-registry-b64d99df4-mtsvs" (UID: "ebd0d655-5116-4a35-89ff-f7575026d4ea") : secret "image-registry-tls" not found Nov 02 13:57:16.701120 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.701122 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls podName:1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d nodeName:}" failed. No retries permitted until 2026-11-02 13:57:17.701109272 +0000 UTC m=+2.884956587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4sfb8" (UID: "1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d") : secret "samples-operator-tls" not found Nov 02 13:57:16.701307 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.701132 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs podName:4441e34f-75c1-40d0-9d06-af34fd3fbb50 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:17.701126962 +0000 UTC m=+2.884974278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs") pod "network-metrics-daemon-r5vdt" (UID: "4441e34f-75c1-40d0-9d06-af34fd3fbb50") : secret "metrics-daemon-secret" not found Nov 02 13:57:16.996762 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.996692 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 13:57:16.997212 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.997153 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/1.log" Nov 02 13:57:16.997212 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.997190 4866 generic.go:356] "Generic (PLEG): container finished" podID="b97f6b80-f53b-4fff-97f6-dff42fb5875f" containerID="f17199c00092221065327692caa5ae2a8d79aa1288c4676e5e9c5aef54b9836d" exitCode=255 Nov 02 13:57:16.997370 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.997263 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" event={"ID":"b97f6b80-f53b-4fff-97f6-dff42fb5875f","Type":"ContainerDied","Data":"f17199c00092221065327692caa5ae2a8d79aa1288c4676e5e9c5aef54b9836d"} Nov 02 13:57:16.997370 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.997314 4866 scope.go:122] "RemoveContainer" containerID="f892c402188f1cfca2d6a95a0b536190c1ff017b4bf04c09813676621b9ec58d" Nov 02 13:57:16.997771 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:16.997751 4866 scope.go:122] "RemoveContainer" containerID="f17199c00092221065327692caa5ae2a8d79aa1288c4676e5e9c5aef54b9836d" Nov 02 13:57:16.997897 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:16.997880 4866 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5hfrp_openshift-console-operator(b97f6b80-f53b-4fff-97f6-dff42fb5875f)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" podUID="b97f6b80-f53b-4fff-97f6-dff42fb5875f" Nov 02 13:57:17.712680 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:17.712637 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:17.712680 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:17.712684 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:57:17.712923 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:17.712716 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:17.712923 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:17.712741 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:17.712923 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:17.712788 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:17.712923 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:17.712795 4866 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 13:57:17.712923 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:17.712828 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:17.712923 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:17.712886 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs podName:4441e34f-75c1-40d0-9d06-af34fd3fbb50 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:19.712865067 +0000 UTC m=+4.896712401 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs") pod "network-metrics-daemon-r5vdt" (UID: "4441e34f-75c1-40d0-9d06-af34fd3fbb50") : secret "metrics-daemon-secret" not found Nov 02 13:57:17.712923 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:17.712904 4866 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 13:57:17.713249 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:17.712983 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:17.713249 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:17.713020 4866 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 13:57:17.713249 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:17.713037 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hvxsl\" (UID: \"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:17.713249 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:17.713062 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs podName:87b5876f-0012-4bdf-b037-25070684476c nodeName:}" failed. No retries permitted until 2026-11-02 13:57:19.713045622 +0000 UTC m=+4.896892950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs") pod "router-default-7899fcdf65-hl87t" (UID: "87b5876f-0012-4bdf-b037-25070684476c") : secret "router-metrics-certs-default" not found Nov 02 13:57:17.713249 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:17.713084 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls podName:ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:19.713074709 +0000 UTC m=+4.896922039 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls") pod "dns-default-cp4vx" (UID: "ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e") : secret "dns-default-metrics-tls" not found Nov 02 13:57:17.713249 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:17.713112 4866 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 13:57:17.713249 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:17.713146 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:17.713249 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:17.713172 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:17.713249 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:17.713185 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert podName:cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:19.713169999 +0000 UTC m=+4.897017337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hvxsl" (UID: "cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1") : secret "networking-console-plugin-cert" not found Nov 02 13:57:17.713249 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:17.713201 4866 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 13:57:17.713249 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:17.713218 4866 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 13:57:17.713249 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:17.713243 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls podName:d32c74f9-dc27-4e9f-a20b-da4c3704acc1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:19.713235937 +0000 UTC m=+4.897083252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x6kpv" (UID: "d32c74f9-dc27-4e9f-a20b-da4c3704acc1") : secret "insights-runtime-extractor-tls" not found Nov 02 13:57:17.713249 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:17.713088 4866 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 13:57:17.713775 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:17.713262 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls podName:b1259f5f-470e-4ca5-8760-37bff1735835 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:19.713247214 +0000 UTC m=+4.897094552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4tgbq" (UID: "b1259f5f-470e-4ca5-8760-37bff1735835") : secret "cluster-monitoring-operator-tls" not found Nov 02 13:57:17.713775 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:17.713613 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87b5876f-0012-4bdf-b037-25070684476c-service-ca-bundle\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:17.715444 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:17.715412 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls\") pod \"image-registry-579f678cff-m54g4\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:17.715525 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:17.715439 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4sfb8\" (UID: \"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:17.715604 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:17.715583 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls\") pod \"image-registry-b64d99df4-mtsvs\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:17.743519 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:17.743488 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:17.744531 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:17.744509 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" Nov 02 13:57:17.750399 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:17.750379 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:17.758564 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:17.758544 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zgxlv_087ad47b-bcc1-4332-9706-8f0ed148252d/dns-node-resolver/0.log" Nov 02 13:57:18.002701 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:18.002629 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 13:57:18.075661 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:18.075628 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8"] Nov 02 13:57:18.079056 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:18.079028 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b64d99df4-mtsvs"] Nov 02 13:57:18.090532 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:18.090507 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-579f678cff-m54g4"] Nov 02 13:57:18.094285 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:18.094240 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd0d655_5116_4a35_89ff_f7575026d4ea.slice/crio-deb41b26ee3c2e629e2249335059d9feea70cbf42787787177991052350e8c53 WatchSource:0}: Error finding container deb41b26ee3c2e629e2249335059d9feea70cbf42787787177991052350e8c53: Status 404 returned error can't find the container with id deb41b26ee3c2e629e2249335059d9feea70cbf42787787177991052350e8c53 Nov 02 13:57:18.094614 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:18.094592 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf212522d_ef61_4b05_aee8_538433003f78.slice/crio-977cddf3fa3f4e9fc148bdc593767996fd6b980eb8115841196bd9b61428f53e WatchSource:0}: Error finding container 977cddf3fa3f4e9fc148bdc593767996fd6b980eb8115841196bd9b61428f53e: Status 404 returned error can't find the container with id 977cddf3fa3f4e9fc148bdc593767996fd6b980eb8115841196bd9b61428f53e Nov 02 13:57:18.116843 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:18.116822 4866 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 13:57:18.356511 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:18.356438 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2sxls_e9b81de7-b765-4eac-afd2-86d99495af3d/node-ca/0.log" Nov 02 13:57:19.011503 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:19.011465 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" event={"ID":"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d","Type":"ContainerStarted","Data":"96c6eb81fb04e3debfb63ad858fd58c68bfebf41524b0a669831e162aaa733d6"} Nov 02 13:57:19.013427 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:19.013347 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" event={"ID":"ebd0d655-5116-4a35-89ff-f7575026d4ea","Type":"ContainerStarted","Data":"e74e9bf9367179efaf8c52f6695c11bbd6a6710c523ec3786561e6bfbe818930"} Nov 02 13:57:19.013552 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:19.013447 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" event={"ID":"ebd0d655-5116-4a35-89ff-f7575026d4ea","Type":"ContainerStarted","Data":"deb41b26ee3c2e629e2249335059d9feea70cbf42787787177991052350e8c53"} Nov 02 13:57:19.015011 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:19.014987 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579f678cff-m54g4" event={"ID":"f212522d-ef61-4b05-aee8-538433003f78","Type":"ContainerStarted","Data":"ec085a665dc3a4b0656fe211f2c134830b5c7ee8055498a0bc0974cec7e56415"} Nov 02 13:57:19.015120 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:19.015029 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579f678cff-m54g4" event={"ID":"f212522d-ef61-4b05-aee8-538433003f78","Type":"ContainerStarted","Data":"977cddf3fa3f4e9fc148bdc593767996fd6b980eb8115841196bd9b61428f53e"} Nov 02 13:57:19.556456 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:19.556423 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f9kv5_7120b45c-1c5b-4c7c-863f-1902f96c00f5/serve-healthcheck-canary/0.log" Nov 02 13:57:19.736279 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:19.736236 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:19.736448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:19.736301 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:57:19.736448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:19.736349 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:19.736448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:19.736387 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:19.736448 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:19.736413 4866 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 13:57:19.736636 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:19.736460 4866 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 13:57:19.736636 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:19.736476 4866 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 13:57:19.736636 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:19.736419 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hvxsl\" (UID: \"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:19.736636 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:19.736510 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls podName:d32c74f9-dc27-4e9f-a20b-da4c3704acc1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:23.736476604 +0000 UTC m=+8.920323941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x6kpv" (UID: "d32c74f9-dc27-4e9f-a20b-da4c3704acc1") : secret "insights-runtime-extractor-tls" not found Nov 02 13:57:19.736636 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:19.736544 4866 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 13:57:19.736636 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:19.736554 4866 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 13:57:19.736636 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:19.736586 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert podName:cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:23.736570543 +0000 UTC m=+8.920417860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hvxsl" (UID: "cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1") : secret "networking-console-plugin-cert" not found Nov 02 13:57:19.736636 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:19.736614 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs podName:4441e34f-75c1-40d0-9d06-af34fd3fbb50 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:23.736599473 +0000 UTC m=+8.920446797 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs") pod "network-metrics-daemon-r5vdt" (UID: "4441e34f-75c1-40d0-9d06-af34fd3fbb50") : secret "metrics-daemon-secret" not found Nov 02 13:57:19.736636 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:19.736629 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls podName:ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:23.736623774 +0000 UTC m=+8.920471094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls") pod "dns-default-cp4vx" (UID: "ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e") : secret "dns-default-metrics-tls" not found Nov 02 13:57:19.737042 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:19.736649 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs podName:87b5876f-0012-4bdf-b037-25070684476c nodeName:}" failed. No retries permitted until 2026-11-02 13:57:23.736644076 +0000 UTC m=+8.920491394 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs") pod "router-default-7899fcdf65-hl87t" (UID: "87b5876f-0012-4bdf-b037-25070684476c") : secret "router-metrics-certs-default" not found Nov 02 13:57:19.737042 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:19.736664 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:19.737042 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:19.736749 4866 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 13:57:19.737042 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:19.736775 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls podName:b1259f5f-470e-4ca5-8760-37bff1735835 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:23.7367687 +0000 UTC m=+8.920616015 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4tgbq" (UID: "b1259f5f-470e-4ca5-8760-37bff1735835") : secret "cluster-monitoring-operator-tls" not found Nov 02 13:57:20.020682 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:20.020639 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" event={"ID":"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d","Type":"ContainerStarted","Data":"c6c9ed3219f36ff458a58f01e60c377374c7b194b536c83bbce1b67c19ee9755"} Nov 02 13:57:20.021134 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:20.020704 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4sfb8" event={"ID":"1b27fbd1-50b5-4ac7-b452-ca5ba5dec66d","Type":"ContainerStarted","Data":"70881a25f75144096976f5e485bbebe21b8b0ab4b9855975d872d06608c9f4bd"} Nov 02 13:57:20.021134 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:20.020663 4866 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 13:57:20.021134 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:20.020804 4866 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 13:57:20.167188 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:20.167115 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm_bd3bd8d9-b081-48bd-be29-582afe6f82fd/kube-storage-version-migrator-operator/0.log" Nov 02 13:57:20.437451 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:20.437373 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:20.491613 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:20.491576 4866 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:20.492012 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:20.491997 4866 scope.go:122] "RemoveContainer" containerID="f17199c00092221065327692caa5ae2a8d79aa1288c4676e5e9c5aef54b9836d" Nov 02 13:57:20.492216 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:20.492194 4866 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5hfrp_openshift-console-operator(b97f6b80-f53b-4fff-97f6-dff42fb5875f)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" podUID="b97f6b80-f53b-4fff-97f6-dff42fb5875f" Nov 02 13:57:22.152842 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:22.152798 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:22.446689 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:22.446658 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:57:23.773546 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:23.773509 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hvxsl\" (UID: \"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:23.773546 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:23.773551 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:23.774182 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:23.773578 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:23.774182 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:23.773632 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:57:23.774182 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:23.773667 4866 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 13:57:23.774182 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:23.773696 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:23.774182 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:23.773734 4866 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 13:57:23.774182 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:23.773739 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert podName:cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:31.77372109 +0000 UTC m=+16.957568440 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hvxsl" (UID: "cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1") : secret "networking-console-plugin-cert" not found Nov 02 13:57:23.774182 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:23.773821 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls podName:d32c74f9-dc27-4e9f-a20b-da4c3704acc1 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:31.773809131 +0000 UTC m=+16.957656447 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x6kpv" (UID: "d32c74f9-dc27-4e9f-a20b-da4c3704acc1") : secret "insights-runtime-extractor-tls" not found Nov 02 13:57:23.774182 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:23.773851 4866 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 13:57:23.774182 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:23.773862 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:23.774182 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:23.773900 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs podName:4441e34f-75c1-40d0-9d06-af34fd3fbb50 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:31.773889081 +0000 UTC m=+16.957736400 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs") pod "network-metrics-daemon-r5vdt" (UID: "4441e34f-75c1-40d0-9d06-af34fd3fbb50") : secret "metrics-daemon-secret" not found Nov 02 13:57:23.774182 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:23.773943 4866 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 13:57:23.774182 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:23.773996 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs podName:87b5876f-0012-4bdf-b037-25070684476c nodeName:}" failed. No retries permitted until 2026-11-02 13:57:31.773983738 +0000 UTC m=+16.957831072 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs") pod "router-default-7899fcdf65-hl87t" (UID: "87b5876f-0012-4bdf-b037-25070684476c") : secret "router-metrics-certs-default" not found Nov 02 13:57:23.776042 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:23.776021 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e-metrics-tls\") pod \"dns-default-cp4vx\" (UID: \"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e\") " pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:23.776125 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:23.776078 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1259f5f-470e-4ca5-8760-37bff1735835-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4tgbq\" (UID: \"b1259f5f-470e-4ca5-8760-37bff1735835\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:24.044455 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:24.044367 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" Nov 02 13:57:24.056517 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:24.056489 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:24.353158 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:24.353077 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq"] Nov 02 13:57:24.358682 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:24.358657 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1259f5f_470e_4ca5_8760_37bff1735835.slice/crio-edd523d4331c93336f3a53f49b92e3214e631d4f02663ea389d668ef076fcafb WatchSource:0}: Error finding container edd523d4331c93336f3a53f49b92e3214e631d4f02663ea389d668ef076fcafb: Status 404 returned error can't find the container with id edd523d4331c93336f3a53f49b92e3214e631d4f02663ea389d668ef076fcafb Nov 02 13:57:24.369583 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:24.369559 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cp4vx"] Nov 02 13:57:24.372692 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:24.372669 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade2f5fa_c7c7_42e8_ad66_80179ef5dd4e.slice/crio-3a02bb1246ad23a4e30f1be195cb523771551fea0c32c11056b0f18ec2d7b64b WatchSource:0}: Error finding container 3a02bb1246ad23a4e30f1be195cb523771551fea0c32c11056b0f18ec2d7b64b: Status 404 returned error can't find the container with id 3a02bb1246ad23a4e30f1be195cb523771551fea0c32c11056b0f18ec2d7b64b Nov 02 13:57:24.796905 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:24.796864 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:24.797416 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:24.797399 4866 scope.go:122] "RemoveContainer" containerID="f17199c00092221065327692caa5ae2a8d79aa1288c4676e5e9c5aef54b9836d" Nov 02 13:57:24.797627 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:24.797607 4866 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5hfrp_openshift-console-operator(b97f6b80-f53b-4fff-97f6-dff42fb5875f)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" podUID="b97f6b80-f53b-4fff-97f6-dff42fb5875f" Nov 02 13:57:25.037403 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:25.037364 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cp4vx" event={"ID":"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e","Type":"ContainerStarted","Data":"3a02bb1246ad23a4e30f1be195cb523771551fea0c32c11056b0f18ec2d7b64b"} Nov 02 13:57:25.038702 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:25.038667 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" event={"ID":"b1259f5f-470e-4ca5-8760-37bff1735835","Type":"ContainerStarted","Data":"edd523d4331c93336f3a53f49b92e3214e631d4f02663ea389d668ef076fcafb"} Nov 02 13:57:25.443870 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:25.443842 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:25.444061 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:25.444048 4866 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 13:57:25.782058 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:25.781978 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:26.047198 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:26.047160 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cp4vx" event={"ID":"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e","Type":"ContainerStarted","Data":"206c9e3ab4d8172f50b9e6d0017ef3968112bc2998cda40fe7a8513b81b3898e"} Nov 02 13:57:26.938669 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:26.938637 4866 scope.go:122] "RemoveContainer" containerID="f17199c00092221065327692caa5ae2a8d79aa1288c4676e5e9c5aef54b9836d" Nov 02 13:57:27.052220 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:27.052195 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 13:57:27.052639 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:27.052292 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" event={"ID":"b97f6b80-f53b-4fff-97f6-dff42fb5875f","Type":"ContainerStarted","Data":"474158d11bc5f3a02eb0636360e0d6a0f7fc2313da95916dba1e8a36ef652d4c"} Nov 02 13:57:27.052639 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:27.052545 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:27.053462 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:27.053432 4866 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-5hfrp container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused" start-of-body= Nov 02 13:57:27.053590 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:27.053488 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" podUID="b97f6b80-f53b-4fff-97f6-dff42fb5875f" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused" Nov 02 13:57:27.054254 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:27.054233 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cp4vx" event={"ID":"ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e","Type":"ContainerStarted","Data":"6dd4f3e61e0892a789646bf201fa469f35ff2175b41cde82614e052320eff108"} Nov 02 13:57:27.054390 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:27.054372 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:27.055686 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:27.055668 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4tgbq" event={"ID":"b1259f5f-470e-4ca5-8760-37bff1735835","Type":"ContainerStarted","Data":"13a9b83af6b4b5acf313efb436c6350bcea201928dbba14be0cd875a275b1715"} Nov 02 13:57:28.064860 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:28.064831 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5hfrp" Nov 02 13:57:31.847791 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:31.847753 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:31.848383 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:31.847819 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:57:31.848383 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:31.847887 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:31.848383 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:31.847923 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hvxsl\" (UID: \"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:31.850238 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:31.850203 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d32c74f9-dc27-4e9f-a20b-da4c3704acc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6kpv\" (UID: \"d32c74f9-dc27-4e9f-a20b-da4c3704acc1\") " pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:31.850363 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:31.850343 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b5876f-0012-4bdf-b037-25070684476c-metrics-certs\") pod \"router-default-7899fcdf65-hl87t\" (UID: \"87b5876f-0012-4bdf-b037-25070684476c\") " pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:31.850594 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:31.850568 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4441e34f-75c1-40d0-9d06-af34fd3fbb50-metrics-certs\") pod \"network-metrics-daemon-r5vdt\" (UID: \"4441e34f-75c1-40d0-9d06-af34fd3fbb50\") " pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:57:31.850659 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:31.850644 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hvxsl\" (UID: \"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:31.857509 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:31.857492 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x6kpv" Nov 02 13:57:32.145614 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:32.145527 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:32.146569 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:32.146546 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" Nov 02 13:57:32.150420 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:32.150398 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5vdt" Nov 02 13:57:32.151321 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:32.151302 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x6kpv"] Nov 02 13:57:32.155809 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:32.155781 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32c74f9_dc27_4e9f_a20b_da4c3704acc1.slice/crio-8b3fd202109efc84470258ef5f8c8e894a5e9a37913b0fa6578e9d43d26ca72b WatchSource:0}: Error finding container 8b3fd202109efc84470258ef5f8c8e894a5e9a37913b0fa6578e9d43d26ca72b: Status 404 returned error can't find the container with id 8b3fd202109efc84470258ef5f8c8e894a5e9a37913b0fa6578e9d43d26ca72b Nov 02 13:57:32.460027 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:32.459994 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-hvxsl"] Nov 02 13:57:32.463478 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:32.463453 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd289ce_fac0_4e37_a9e8_c8b1cc8fa4c1.slice/crio-31c3db5ef51d47d74aefa010b199d1cd14924f5c186efb9e84ebf1f21ee1ad6a WatchSource:0}: Error finding container 31c3db5ef51d47d74aefa010b199d1cd14924f5c186efb9e84ebf1f21ee1ad6a: Status 404 returned error can't find the container with id 31c3db5ef51d47d74aefa010b199d1cd14924f5c186efb9e84ebf1f21ee1ad6a Nov 02 13:57:32.465535 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:32.465511 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7899fcdf65-hl87t"] Nov 02 13:57:32.468824 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:32.468803 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b5876f_0012_4bdf_b037_25070684476c.slice/crio-ebbd6decaa3b224538d49846ce779543627eea63c5bfad42cc49d8ad22f0ed4d WatchSource:0}: Error finding container ebbd6decaa3b224538d49846ce779543627eea63c5bfad42cc49d8ad22f0ed4d: Status 404 returned error can't find the container with id ebbd6decaa3b224538d49846ce779543627eea63c5bfad42cc49d8ad22f0ed4d Nov 02 13:57:32.479050 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:32.479025 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r5vdt"] Nov 02 13:57:32.481885 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:32.481860 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4441e34f_75c1_40d0_9d06_af34fd3fbb50.slice/crio-735e004c26052602ffae80531d6dd5c9c5c4760a255556abc22b7c3d6cb75698 WatchSource:0}: Error finding container 735e004c26052602ffae80531d6dd5c9c5c4760a255556abc22b7c3d6cb75698: Status 404 returned error can't find the container with id 735e004c26052602ffae80531d6dd5c9c5c4760a255556abc22b7c3d6cb75698 Nov 02 13:57:32.675984 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:32.675934 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-579f678cff-m54g4"] Nov 02 13:57:33.078158 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:33.078113 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r5vdt" event={"ID":"4441e34f-75c1-40d0-9d06-af34fd3fbb50","Type":"ContainerStarted","Data":"735e004c26052602ffae80531d6dd5c9c5c4760a255556abc22b7c3d6cb75698"} Nov 02 13:57:33.081105 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:33.081066 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7899fcdf65-hl87t" event={"ID":"87b5876f-0012-4bdf-b037-25070684476c","Type":"ContainerStarted","Data":"1b1925348f1146cca2358554e540862aa0c269b2f5d5ac9fef934eb1d4a5a223"} Nov 02 13:57:33.081225 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:33.081130 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7899fcdf65-hl87t" event={"ID":"87b5876f-0012-4bdf-b037-25070684476c","Type":"ContainerStarted","Data":"ebbd6decaa3b224538d49846ce779543627eea63c5bfad42cc49d8ad22f0ed4d"} Nov 02 13:57:33.083118 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:33.083089 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" event={"ID":"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1","Type":"ContainerStarted","Data":"31c3db5ef51d47d74aefa010b199d1cd14924f5c186efb9e84ebf1f21ee1ad6a"} Nov 02 13:57:33.085083 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:33.085057 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x6kpv" event={"ID":"d32c74f9-dc27-4e9f-a20b-da4c3704acc1","Type":"ContainerStarted","Data":"a8fe4b15bd7bb4d8b67c20a377d1925d8d6e663e744324cdd32a9c4995449194"} Nov 02 13:57:33.085169 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:33.085114 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x6kpv" event={"ID":"d32c74f9-dc27-4e9f-a20b-da4c3704acc1","Type":"ContainerStarted","Data":"8b3fd202109efc84470258ef5f8c8e894a5e9a37913b0fa6578e9d43d26ca72b"} Nov 02 13:57:33.145796 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:33.145758 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:33.149108 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:33.149079 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:34.090438 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:34.090402 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-hvxsl" event={"ID":"cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1","Type":"ContainerStarted","Data":"fda1989f24c6c430e77ef194cedd3afb5f03a7b6c6d624b8b8515c003f91be4d"} Nov 02 13:57:34.092484 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:34.092455 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x6kpv" event={"ID":"d32c74f9-dc27-4e9f-a20b-da4c3704acc1","Type":"ContainerStarted","Data":"d1a1a5a7a5c87d821fc21a2ff25a62b5b60938cd0d1f0e46dc47e0ffdfd0c7d3"} Nov 02 13:57:34.094409 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:34.094383 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r5vdt" event={"ID":"4441e34f-75c1-40d0-9d06-af34fd3fbb50","Type":"ContainerStarted","Data":"99770ad469cbe2b49682aea0edcdf8860dacce28c173ba3bbcbb39682e621a10"} Nov 02 13:57:34.094525 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:34.094476 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:34.094525 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:34.094513 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r5vdt" event={"ID":"4441e34f-75c1-40d0-9d06-af34fd3fbb50","Type":"ContainerStarted","Data":"6bfcaf0a341785ffbd5d572440699c9e7f4c4480a1bbd60f3934a9dcae2153ca"} Nov 02 13:57:34.095799 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:34.095780 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7899fcdf65-hl87t" Nov 02 13:57:34.106503 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:34.106482 4866 patch_prober.go:29] interesting pod/image-registry-b64d99df4-mtsvs container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 13:57:34.106590 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:34.106524 4866 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" podUID="ebd0d655-5116-4a35-89ff-f7575026d4ea" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 13:57:35.100378 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:35.100336 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x6kpv" event={"ID":"d32c74f9-dc27-4e9f-a20b-da4c3704acc1","Type":"ContainerStarted","Data":"bf11274d6440e274af69bed7a2e0700e16a7b7f8593fb6c409d586a702198f88"} Nov 02 13:57:35.125875 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:35.125841 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jg9lw" Nov 02 13:57:36.008043 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:36.007998 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 13:57:37.060583 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:37.060551 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cp4vx" Nov 02 13:57:40.287490 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.287460 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hd8x9"] Nov 02 13:57:40.305697 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.305671 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.364625 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.364581 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 13:57:40.364625 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.364605 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 13:57:40.364860 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.364640 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-z8rpr\"" Nov 02 13:57:40.398911 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.398882 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 13:57:40.399094 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.398982 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 13:57:40.420244 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.420206 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a80e1d69-fa23-46db-88f9-18fd14912fa8-node-exporter-accelerators-collector-config\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.420389 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.420254 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a80e1d69-fa23-46db-88f9-18fd14912fa8-node-exporter-tls\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.420389 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.420328 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a80e1d69-fa23-46db-88f9-18fd14912fa8-root\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.420456 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.420401 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a80e1d69-fa23-46db-88f9-18fd14912fa8-metrics-client-ca\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.420456 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.420419 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a80e1d69-fa23-46db-88f9-18fd14912fa8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.420456 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.420442 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a80e1d69-fa23-46db-88f9-18fd14912fa8-node-exporter-wtmp\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.420547 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.420522 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a80e1d69-fa23-46db-88f9-18fd14912fa8-node-exporter-textfile\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.420579 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.420563 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpxn6\" (UniqueName: \"kubernetes.io/projected/a80e1d69-fa23-46db-88f9-18fd14912fa8-kube-api-access-fpxn6\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.420610 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.420601 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a80e1d69-fa23-46db-88f9-18fd14912fa8-sys\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.521602 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.521565 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a80e1d69-fa23-46db-88f9-18fd14912fa8-node-exporter-textfile\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.521602 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.521604 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpxn6\" (UniqueName: \"kubernetes.io/projected/a80e1d69-fa23-46db-88f9-18fd14912fa8-kube-api-access-fpxn6\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.521916 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.521626 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a80e1d69-fa23-46db-88f9-18fd14912fa8-sys\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.521916 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.521650 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a80e1d69-fa23-46db-88f9-18fd14912fa8-node-exporter-accelerators-collector-config\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.521916 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.521688 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a80e1d69-fa23-46db-88f9-18fd14912fa8-node-exporter-tls\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.521916 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.521720 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a80e1d69-fa23-46db-88f9-18fd14912fa8-sys\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.521916 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.521727 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a80e1d69-fa23-46db-88f9-18fd14912fa8-root\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.521916 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.521769 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a80e1d69-fa23-46db-88f9-18fd14912fa8-root\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.521916 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.521805 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a80e1d69-fa23-46db-88f9-18fd14912fa8-metrics-client-ca\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.521916 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.521833 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a80e1d69-fa23-46db-88f9-18fd14912fa8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.521916 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.521869 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a80e1d69-fa23-46db-88f9-18fd14912fa8-node-exporter-wtmp\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.522430 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.522016 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a80e1d69-fa23-46db-88f9-18fd14912fa8-node-exporter-textfile\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.522430 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.522045 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a80e1d69-fa23-46db-88f9-18fd14912fa8-node-exporter-wtmp\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.522430 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.522336 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a80e1d69-fa23-46db-88f9-18fd14912fa8-node-exporter-accelerators-collector-config\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.522430 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.522403 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a80e1d69-fa23-46db-88f9-18fd14912fa8-metrics-client-ca\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.524171 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.524151 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a80e1d69-fa23-46db-88f9-18fd14912fa8-node-exporter-tls\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.524263 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.524152 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a80e1d69-fa23-46db-88f9-18fd14912fa8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.545668 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.545596 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpxn6\" (UniqueName: \"kubernetes.io/projected/a80e1d69-fa23-46db-88f9-18fd14912fa8-kube-api-access-fpxn6\") pod \"node-exporter-hd8x9\" (UID: \"a80e1d69-fa23-46db-88f9-18fd14912fa8\") " pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.656085 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:40.656047 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hd8x9" Nov 02 13:57:40.699133 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:57:40.699100 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80e1d69_fa23_46db_88f9_18fd14912fa8.slice/crio-ae91bac903baba226e33a2d87a9fa661ace4275dc3e6ca6d7dffcdfbd28d4e70 WatchSource:0}: Error finding container ae91bac903baba226e33a2d87a9fa661ace4275dc3e6ca6d7dffcdfbd28d4e70: Status 404 returned error can't find the container with id ae91bac903baba226e33a2d87a9fa661ace4275dc3e6ca6d7dffcdfbd28d4e70 Nov 02 13:57:41.123798 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:41.123759 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hd8x9" event={"ID":"a80e1d69-fa23-46db-88f9-18fd14912fa8","Type":"ContainerStarted","Data":"ae91bac903baba226e33a2d87a9fa661ace4275dc3e6ca6d7dffcdfbd28d4e70"} Nov 02 13:57:42.156812 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:42.156785 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:57:42.449798 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:42.449763 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pstlx" Nov 02 13:57:42.682053 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:42.682026 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:43.137564 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:43.137528 4866 generic.go:356] "Generic (PLEG): container finished" podID="a80e1d69-fa23-46db-88f9-18fd14912fa8" containerID="987833fd1f7bcc60574d5df6b09bfbc19c290815f41e806e63ace9d9209b3582" exitCode=0 Nov 02 13:57:43.137763 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:43.137574 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hd8x9" event={"ID":"a80e1d69-fa23-46db-88f9-18fd14912fa8","Type":"ContainerDied","Data":"987833fd1f7bcc60574d5df6b09bfbc19c290815f41e806e63ace9d9209b3582"} Nov 02 13:57:43.684836 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:43.684805 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5vmgp" Nov 02 13:57:44.143442 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:44.143409 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hd8x9" event={"ID":"a80e1d69-fa23-46db-88f9-18fd14912fa8","Type":"ContainerStarted","Data":"4b012bd8b90e5b32b599e7c98d4206a6b82eaa23f2d0f1d99438a449828153a7"} Nov 02 13:57:44.143615 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:44.143464 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hd8x9" event={"ID":"a80e1d69-fa23-46db-88f9-18fd14912fa8","Type":"ContainerStarted","Data":"f455b45a736e6a34f10519093ea0f92cefd83968e695718d94ca2364384272e0"} Nov 02 13:57:44.202554 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:44.202501 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hd8x9" podStartSLOduration=2.749383279 podStartE2EDuration="4.202482856s" totalImagesPullingTime="1.380983263s" totalInitContainerRuntime="72.116314ms" isStatefulPod=true podCreationTimestamp="2026-11-02 13:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:44.201558383 +0000 UTC m=+29.385405720" watchObservedRunningTime="2026-11-02 13:57:44.202482856 +0000 UTC m=+29.386330184" Nov 02 13:57:54.789806 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:54.789768 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b64d99df4-mtsvs"] Nov 02 13:57:57.698558 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:57.698445 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-579f678cff-m54g4" podUID="f212522d-ef61-4b05-aee8-538433003f78" containerName="registry" containerID="cri-o://ec085a665dc3a4b0656fe211f2c134830b5c7ee8055498a0bc0974cec7e56415" gracePeriod=30 Nov 02 13:57:57.951613 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:57.951555 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:58.070235 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.070201 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-trusted-ca\") pod \"f212522d-ef61-4b05-aee8-538433003f78\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " Nov 02 13:57:58.070419 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.070245 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls\") pod \"f212522d-ef61-4b05-aee8-538433003f78\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " Nov 02 13:57:58.070419 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.070277 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-registry-certificates\") pod \"f212522d-ef61-4b05-aee8-538433003f78\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " Nov 02 13:57:58.070419 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.070306 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-kube-api-access-ghtzt\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-kube-api-access-ghtzt\") pod \"f212522d-ef61-4b05-aee8-538433003f78\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " Nov 02 13:57:58.070419 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.070333 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-image-registry-private-configuration\") pod \"f212522d-ef61-4b05-aee8-538433003f78\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " Nov 02 13:57:58.070419 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.070360 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-installation-pull-secrets\") pod \"f212522d-ef61-4b05-aee8-538433003f78\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " Nov 02 13:57:58.070419 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.070394 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f212522d-ef61-4b05-aee8-538433003f78-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f212522d-ef61-4b05-aee8-538433003f78-ca-trust-extracted\") pod \"f212522d-ef61-4b05-aee8-538433003f78\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " Nov 02 13:57:58.070739 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.070447 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-bound-sa-token\") pod \"f212522d-ef61-4b05-aee8-538433003f78\" (UID: \"f212522d-ef61-4b05-aee8-538433003f78\") " Nov 02 13:57:58.070739 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.070700 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-trusted-ca" pod "f212522d-ef61-4b05-aee8-538433003f78" (UID: "f212522d-ef61-4b05-aee8-538433003f78"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:57:58.070844 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.070743 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-registry-certificates" pod "f212522d-ef61-4b05-aee8-538433003f78" (UID: "f212522d-ef61-4b05-aee8-538433003f78"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:57:58.073024 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.072976 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-kube-api-access-ghtzt" pod "f212522d-ef61-4b05-aee8-538433003f78" (UID: "f212522d-ef61-4b05-aee8-538433003f78"). InnerVolumeSpecName "kube-api-access-ghtzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:57:58.073156 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.073031 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-image-registry-private-configuration" pod "f212522d-ef61-4b05-aee8-538433003f78" (UID: "f212522d-ef61-4b05-aee8-538433003f78"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:57:58.073156 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.073063 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls" pod "f212522d-ef61-4b05-aee8-538433003f78" (UID: "f212522d-ef61-4b05-aee8-538433003f78"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:57:58.073274 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.073147 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-bound-sa-token" pod "f212522d-ef61-4b05-aee8-538433003f78" (UID: "f212522d-ef61-4b05-aee8-538433003f78"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:57:58.073274 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.073199 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-installation-pull-secrets" pod "f212522d-ef61-4b05-aee8-538433003f78" (UID: "f212522d-ef61-4b05-aee8-538433003f78"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:57:58.077097 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.077068 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f212522d-ef61-4b05-aee8-538433003f78-ca-trust-extracted" pod "f212522d-ef61-4b05-aee8-538433003f78" (UID: "f212522d-ef61-4b05-aee8-538433003f78"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:57:58.171667 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.171633 4866 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f212522d-ef61-4b05-aee8-538433003f78-ca-trust-extracted\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:57:58.171667 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.171661 4866 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-bound-sa-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:57:58.171667 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.171671 4866 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-trusted-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:57:58.171884 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.171679 4866 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-registry-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:57:58.171884 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.171688 4866 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f212522d-ef61-4b05-aee8-538433003f78-registry-certificates\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:57:58.171884 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.171697 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghtzt\" (UniqueName: \"kubernetes.io/projected/f212522d-ef61-4b05-aee8-538433003f78-kube-api-access-ghtzt\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:57:58.171884 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.171707 4866 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-image-registry-private-configuration\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:57:58.171884 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.171716 4866 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f212522d-ef61-4b05-aee8-538433003f78-installation-pull-secrets\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:57:58.196143 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.196107 4866 generic.go:356] "Generic (PLEG): container finished" podID="f212522d-ef61-4b05-aee8-538433003f78" containerID="ec085a665dc3a4b0656fe211f2c134830b5c7ee8055498a0bc0974cec7e56415" exitCode=0 Nov 02 13:57:58.196304 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.196156 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579f678cff-m54g4" Nov 02 13:57:58.196304 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.196195 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579f678cff-m54g4" event={"ID":"f212522d-ef61-4b05-aee8-538433003f78","Type":"ContainerDied","Data":"ec085a665dc3a4b0656fe211f2c134830b5c7ee8055498a0bc0974cec7e56415"} Nov 02 13:57:58.196304 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.196249 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579f678cff-m54g4" event={"ID":"f212522d-ef61-4b05-aee8-538433003f78","Type":"ContainerDied","Data":"977cddf3fa3f4e9fc148bdc593767996fd6b980eb8115841196bd9b61428f53e"} Nov 02 13:57:58.196304 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.196269 4866 scope.go:122] "RemoveContainer" containerID="ec085a665dc3a4b0656fe211f2c134830b5c7ee8055498a0bc0974cec7e56415" Nov 02 13:57:58.205171 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.205152 4866 scope.go:122] "RemoveContainer" containerID="ec085a665dc3a4b0656fe211f2c134830b5c7ee8055498a0bc0974cec7e56415" Nov 02 13:57:58.205453 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:57:58.205435 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec085a665dc3a4b0656fe211f2c134830b5c7ee8055498a0bc0974cec7e56415\": container with ID starting with ec085a665dc3a4b0656fe211f2c134830b5c7ee8055498a0bc0974cec7e56415 not found: ID does not exist" containerID="ec085a665dc3a4b0656fe211f2c134830b5c7ee8055498a0bc0974cec7e56415" Nov 02 13:57:58.205522 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.205465 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec085a665dc3a4b0656fe211f2c134830b5c7ee8055498a0bc0974cec7e56415"} err="failed to get container status \"ec085a665dc3a4b0656fe211f2c134830b5c7ee8055498a0bc0974cec7e56415\": rpc error: code = NotFound desc = could not find container \"ec085a665dc3a4b0656fe211f2c134830b5c7ee8055498a0bc0974cec7e56415\": container with ID starting with ec085a665dc3a4b0656fe211f2c134830b5c7ee8055498a0bc0974cec7e56415 not found: ID does not exist" Nov 02 13:57:58.248479 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.248450 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-579f678cff-m54g4"] Nov 02 13:57:58.254530 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.254498 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-579f678cff-m54g4"] Nov 02 13:57:58.943964 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:57:58.943915 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f212522d-ef61-4b05-aee8-538433003f78" path="/var/lib/kubelet/pods/f212522d-ef61-4b05-aee8-538433003f78/volumes" Nov 02 13:58:06.008819 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:58:06.008780 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 13:58:19.809512 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:19.809465 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" podUID="ebd0d655-5116-4a35-89ff-f7575026d4ea" containerName="registry" containerID="cri-o://e74e9bf9367179efaf8c52f6695c11bbd6a6710c523ec3786561e6bfbe818930" gracePeriod=30 Nov 02 13:58:20.057806 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.057779 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:58:20.137330 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.137251 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-installation-pull-secrets\") pod \"ebd0d655-5116-4a35-89ff-f7575026d4ea\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " Nov 02 13:58:20.137330 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.137309 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-image-registry-private-configuration\") pod \"ebd0d655-5116-4a35-89ff-f7575026d4ea\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " Nov 02 13:58:20.137494 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.137370 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-certificates\") pod \"ebd0d655-5116-4a35-89ff-f7575026d4ea\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " Nov 02 13:58:20.137494 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.137395 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd0d655-5116-4a35-89ff-f7575026d4ea-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebd0d655-5116-4a35-89ff-f7575026d4ea-ca-trust-extracted\") pod \"ebd0d655-5116-4a35-89ff-f7575026d4ea\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " Nov 02 13:58:20.137494 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.137417 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-bound-sa-token\") pod \"ebd0d655-5116-4a35-89ff-f7575026d4ea\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " Nov 02 13:58:20.137494 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.137439 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-trusted-ca\") pod \"ebd0d655-5116-4a35-89ff-f7575026d4ea\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " Nov 02 13:58:20.137494 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.137468 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls\") pod \"ebd0d655-5116-4a35-89ff-f7575026d4ea\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " Nov 02 13:58:20.137703 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.137495 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-kube-api-access-75khg\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-kube-api-access-75khg\") pod \"ebd0d655-5116-4a35-89ff-f7575026d4ea\" (UID: \"ebd0d655-5116-4a35-89ff-f7575026d4ea\") " Nov 02 13:58:20.138207 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.138152 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-trusted-ca" pod "ebd0d655-5116-4a35-89ff-f7575026d4ea" (UID: "ebd0d655-5116-4a35-89ff-f7575026d4ea"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:58:20.138207 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.138172 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-certificates" pod "ebd0d655-5116-4a35-89ff-f7575026d4ea" (UID: "ebd0d655-5116-4a35-89ff-f7575026d4ea"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:58:20.140448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.140405 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-bound-sa-token" pod "ebd0d655-5116-4a35-89ff-f7575026d4ea" (UID: "ebd0d655-5116-4a35-89ff-f7575026d4ea"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:58:20.140448 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.140424 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-installation-pull-secrets" pod "ebd0d655-5116-4a35-89ff-f7575026d4ea" (UID: "ebd0d655-5116-4a35-89ff-f7575026d4ea"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:20.140603 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.140487 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-kube-api-access-75khg" pod "ebd0d655-5116-4a35-89ff-f7575026d4ea" (UID: "ebd0d655-5116-4a35-89ff-f7575026d4ea"). InnerVolumeSpecName "kube-api-access-75khg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:58:20.140603 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.140536 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls" pod "ebd0d655-5116-4a35-89ff-f7575026d4ea" (UID: "ebd0d655-5116-4a35-89ff-f7575026d4ea"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:58:20.140970 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.140930 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-image-registry-private-configuration" pod "ebd0d655-5116-4a35-89ff-f7575026d4ea" (UID: "ebd0d655-5116-4a35-89ff-f7575026d4ea"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:20.144768 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.144705 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd0d655-5116-4a35-89ff-f7575026d4ea-ca-trust-extracted" pod "ebd0d655-5116-4a35-89ff-f7575026d4ea" (UID: "ebd0d655-5116-4a35-89ff-f7575026d4ea"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:58:20.238512 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.238471 4866 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:58:20.238512 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.238504 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75khg\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-kube-api-access-75khg\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:58:20.238512 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.238515 4866 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-installation-pull-secrets\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:58:20.238732 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.238526 4866 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebd0d655-5116-4a35-89ff-f7575026d4ea-image-registry-private-configuration\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:58:20.238732 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.238537 4866 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-registry-certificates\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:58:20.238732 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.238546 4866 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebd0d655-5116-4a35-89ff-f7575026d4ea-ca-trust-extracted\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:58:20.238732 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.238554 4866 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebd0d655-5116-4a35-89ff-f7575026d4ea-bound-sa-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:58:20.238732 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.238562 4866 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebd0d655-5116-4a35-89ff-f7575026d4ea-trusted-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:58:20.276614 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.276582 4866 generic.go:356] "Generic (PLEG): container finished" podID="ebd0d655-5116-4a35-89ff-f7575026d4ea" containerID="e74e9bf9367179efaf8c52f6695c11bbd6a6710c523ec3786561e6bfbe818930" exitCode=0 Nov 02 13:58:20.276762 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.276630 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" Nov 02 13:58:20.276762 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.276626 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" event={"ID":"ebd0d655-5116-4a35-89ff-f7575026d4ea","Type":"ContainerDied","Data":"e74e9bf9367179efaf8c52f6695c11bbd6a6710c523ec3786561e6bfbe818930"} Nov 02 13:58:20.276966 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.276762 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b64d99df4-mtsvs" event={"ID":"ebd0d655-5116-4a35-89ff-f7575026d4ea","Type":"ContainerDied","Data":"deb41b26ee3c2e629e2249335059d9feea70cbf42787787177991052350e8c53"} Nov 02 13:58:20.276966 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.276796 4866 scope.go:122] "RemoveContainer" containerID="e74e9bf9367179efaf8c52f6695c11bbd6a6710c523ec3786561e6bfbe818930" Nov 02 13:58:20.284918 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.284902 4866 scope.go:122] "RemoveContainer" containerID="e74e9bf9367179efaf8c52f6695c11bbd6a6710c523ec3786561e6bfbe818930" Nov 02 13:58:20.285182 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:58:20.285166 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e74e9bf9367179efaf8c52f6695c11bbd6a6710c523ec3786561e6bfbe818930\": container with ID starting with e74e9bf9367179efaf8c52f6695c11bbd6a6710c523ec3786561e6bfbe818930 not found: ID does not exist" containerID="e74e9bf9367179efaf8c52f6695c11bbd6a6710c523ec3786561e6bfbe818930" Nov 02 13:58:20.285226 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.285191 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e74e9bf9367179efaf8c52f6695c11bbd6a6710c523ec3786561e6bfbe818930"} err="failed to get container status \"e74e9bf9367179efaf8c52f6695c11bbd6a6710c523ec3786561e6bfbe818930\": rpc error: code = NotFound desc = could not find container \"e74e9bf9367179efaf8c52f6695c11bbd6a6710c523ec3786561e6bfbe818930\": container with ID starting with e74e9bf9367179efaf8c52f6695c11bbd6a6710c523ec3786561e6bfbe818930 not found: ID does not exist" Nov 02 13:58:20.338333 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.335532 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b64d99df4-mtsvs"] Nov 02 13:58:20.340226 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.340194 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b64d99df4-mtsvs"] Nov 02 13:58:20.943853 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:20.943811 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebd0d655-5116-4a35-89ff-f7575026d4ea" path="/var/lib/kubelet/pods/ebd0d655-5116-4a35-89ff-f7575026d4ea/volumes" Nov 02 13:58:30.313717 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:30.313680 4866 generic.go:356] "Generic (PLEG): container finished" podID="4f856415-9f37-415c-8dd5-65fc1ef08abc" containerID="7951888861c1f32310bc8a232a5ab2e557ce8d3f55ed43771df23a2b98cc7fe2" exitCode=0 Nov 02 13:58:30.314230 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:30.313753 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fg7tg" event={"ID":"4f856415-9f37-415c-8dd5-65fc1ef08abc","Type":"ContainerDied","Data":"7951888861c1f32310bc8a232a5ab2e557ce8d3f55ed43771df23a2b98cc7fe2"} Nov 02 13:58:30.314230 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:30.314089 4866 scope.go:122] "RemoveContainer" containerID="7951888861c1f32310bc8a232a5ab2e557ce8d3f55ed43771df23a2b98cc7fe2" Nov 02 13:58:30.315297 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:30.315278 4866 generic.go:356] "Generic (PLEG): container finished" podID="fc0287f6-d102-43a4-80cd-62b3acfb3991" containerID="35927388778ae1f3abd2254eb630d4e33699f4d07251b4c41f446dcdafef0de3" exitCode=0 Nov 02 13:58:30.315369 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:30.315349 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" event={"ID":"fc0287f6-d102-43a4-80cd-62b3acfb3991","Type":"ContainerDied","Data":"35927388778ae1f3abd2254eb630d4e33699f4d07251b4c41f446dcdafef0de3"} Nov 02 13:58:30.315639 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:30.315615 4866 scope.go:122] "RemoveContainer" containerID="35927388778ae1f3abd2254eb630d4e33699f4d07251b4c41f446dcdafef0de3" Nov 02 13:58:30.908566 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:30.908528 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-bpfbx-bmlmm"] Nov 02 13:58:30.908911 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:30.908896 4866 cpu_manager.go:414] "containerMap: removing container" podUID="f212522d-ef61-4b05-aee8-538433003f78" containerName="registry" Nov 02 13:58:30.908997 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:30.908911 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f212522d-ef61-4b05-aee8-538433003f78" containerName="registry" Nov 02 13:58:30.908997 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:30.908923 4866 cpu_manager.go:414] "containerMap: removing container" podUID="ebd0d655-5116-4a35-89ff-f7575026d4ea" containerName="registry" Nov 02 13:58:30.908997 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:30.908928 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebd0d655-5116-4a35-89ff-f7575026d4ea" containerName="registry" Nov 02 13:58:30.908997 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:30.908994 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="f212522d-ef61-4b05-aee8-538433003f78" containerName="registry" Nov 02 13:58:30.909180 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:30.909002 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebd0d655-5116-4a35-89ff-f7575026d4ea" containerName="registry" Nov 02 13:58:30.911960 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:30.911932 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" Nov 02 13:58:30.933671 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:30.933644 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bpfbx-bmlmm"] Nov 02 13:58:31.015498 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:31.015457 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wrmc\" (UniqueName: \"kubernetes.io/projected/56375571-cada-4ce6-a1b7-54be99ab9904-kube-api-access-7wrmc\") pod \"periodic-gathering-bpfbx-bmlmm\" (UID: \"56375571-cada-4ce6-a1b7-54be99ab9904\") " pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" Nov 02 13:58:31.015498 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:31.015498 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/56375571-cada-4ce6-a1b7-54be99ab9904-archives-path\") pod \"periodic-gathering-bpfbx-bmlmm\" (UID: \"56375571-cada-4ce6-a1b7-54be99ab9904\") " pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" Nov 02 13:58:31.015715 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:31.015569 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56375571-cada-4ce6-a1b7-54be99ab9904-service-ca-bundle\") pod \"periodic-gathering-bpfbx-bmlmm\" (UID: \"56375571-cada-4ce6-a1b7-54be99ab9904\") " pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" Nov 02 13:58:31.015715 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:31.015614 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56375571-cada-4ce6-a1b7-54be99ab9904-trusted-ca-bundle\") pod \"periodic-gathering-bpfbx-bmlmm\" (UID: \"56375571-cada-4ce6-a1b7-54be99ab9904\") " pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" Nov 02 13:58:31.116147 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:31.116113 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wrmc\" (UniqueName: \"kubernetes.io/projected/56375571-cada-4ce6-a1b7-54be99ab9904-kube-api-access-7wrmc\") pod \"periodic-gathering-bpfbx-bmlmm\" (UID: \"56375571-cada-4ce6-a1b7-54be99ab9904\") " pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" Nov 02 13:58:31.116339 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:31.116154 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/56375571-cada-4ce6-a1b7-54be99ab9904-archives-path\") pod \"periodic-gathering-bpfbx-bmlmm\" (UID: \"56375571-cada-4ce6-a1b7-54be99ab9904\") " pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" Nov 02 13:58:31.116339 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:31.116223 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56375571-cada-4ce6-a1b7-54be99ab9904-service-ca-bundle\") pod \"periodic-gathering-bpfbx-bmlmm\" (UID: \"56375571-cada-4ce6-a1b7-54be99ab9904\") " pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" Nov 02 13:58:31.116455 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:31.116363 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56375571-cada-4ce6-a1b7-54be99ab9904-trusted-ca-bundle\") pod \"periodic-gathering-bpfbx-bmlmm\" (UID: \"56375571-cada-4ce6-a1b7-54be99ab9904\") " pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" Nov 02 13:58:31.116645 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:31.116621 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/56375571-cada-4ce6-a1b7-54be99ab9904-archives-path\") pod \"periodic-gathering-bpfbx-bmlmm\" (UID: \"56375571-cada-4ce6-a1b7-54be99ab9904\") " pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" Nov 02 13:58:31.116913 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:31.116894 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56375571-cada-4ce6-a1b7-54be99ab9904-service-ca-bundle\") pod \"periodic-gathering-bpfbx-bmlmm\" (UID: \"56375571-cada-4ce6-a1b7-54be99ab9904\") " pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" Nov 02 13:58:31.117066 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:31.117051 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56375571-cada-4ce6-a1b7-54be99ab9904-trusted-ca-bundle\") pod \"periodic-gathering-bpfbx-bmlmm\" (UID: \"56375571-cada-4ce6-a1b7-54be99ab9904\") " pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" Nov 02 13:58:31.139841 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:31.139804 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wrmc\" (UniqueName: \"kubernetes.io/projected/56375571-cada-4ce6-a1b7-54be99ab9904-kube-api-access-7wrmc\") pod \"periodic-gathering-bpfbx-bmlmm\" (UID: \"56375571-cada-4ce6-a1b7-54be99ab9904\") " pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" Nov 02 13:58:31.223691 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:31.223652 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" Nov 02 13:58:31.321763 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:31.321733 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fg7tg" event={"ID":"4f856415-9f37-415c-8dd5-65fc1ef08abc","Type":"ContainerStarted","Data":"8dc871c6c3a3e4cf4f3c088d734d4fa9ddfba807ce9c84f7d67f7ee6b13c49e9"} Nov 02 13:58:31.323574 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:31.323551 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zv6s5" event={"ID":"fc0287f6-d102-43a4-80cd-62b3acfb3991","Type":"ContainerStarted","Data":"0ae0148be361c1d6117a470e58e5f35e0f0d53bf4154d1376c423d1fdb63cca6"} Nov 02 13:58:31.519663 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:31.519230 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bpfbx-bmlmm"] Nov 02 13:58:31.521894 ip-10-0-134-99 kubenswrapper[4866]: W1102 13:58:31.521868 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56375571_cada_4ce6_a1b7_54be99ab9904.slice/crio-43c1d03e71239afd9b77c801f6ef1c8b3d07735f90a9950d28206a2aa860c11e WatchSource:0}: Error finding container 43c1d03e71239afd9b77c801f6ef1c8b3d07735f90a9950d28206a2aa860c11e: Status 404 returned error can't find the container with id 43c1d03e71239afd9b77c801f6ef1c8b3d07735f90a9950d28206a2aa860c11e Nov 02 13:58:32.329192 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:32.329152 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" event={"ID":"56375571-cada-4ce6-a1b7-54be99ab9904","Type":"ContainerStarted","Data":"c630b1859c5975fab709080036f20cb3411377757e911533277a5d5f5af356a1"} Nov 02 13:58:32.329614 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:32.329206 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" event={"ID":"56375571-cada-4ce6-a1b7-54be99ab9904","Type":"ContainerStarted","Data":"43c1d03e71239afd9b77c801f6ef1c8b3d07735f90a9950d28206a2aa860c11e"} Nov 02 13:58:32.370056 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:32.370004 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" podStartSLOduration=2.369990957 podStartE2EDuration="2.369990957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:32.369096673 +0000 UTC m=+77.552944009" watchObservedRunningTime="2026-11-02 13:58:32.369990957 +0000 UTC m=+77.553838296" Nov 02 13:58:36.008710 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:58:36.008673 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 13:58:40.363212 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:40.363179 4866 generic.go:356] "Generic (PLEG): container finished" podID="bd3bd8d9-b081-48bd-be29-582afe6f82fd" containerID="67d26bae59a994616fbc2d460aa9ee0a8042ac478a497572196838e94da65966" exitCode=0 Nov 02 13:58:40.363690 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:40.363226 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" event={"ID":"bd3bd8d9-b081-48bd-be29-582afe6f82fd","Type":"ContainerDied","Data":"67d26bae59a994616fbc2d460aa9ee0a8042ac478a497572196838e94da65966"} Nov 02 13:58:40.363690 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:40.363526 4866 scope.go:122] "RemoveContainer" containerID="67d26bae59a994616fbc2d460aa9ee0a8042ac478a497572196838e94da65966" Nov 02 13:58:41.368780 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:58:41.368745 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tgwdm" event={"ID":"bd3bd8d9-b081-48bd-be29-582afe6f82fd","Type":"ContainerStarted","Data":"19438d9c7e6965a6d15998b2b0f893c3e1293ae2f08d75f6e7085dabc0ea1d9c"} Nov 02 13:59:06.008744 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:59:06.008705 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 13:59:36.010658 ip-10-0-134-99 kubenswrapper[4866]: E1102 13:59:36.010565 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 13:59:55.636309 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:59:55.636273 4866 generic.go:356] "Generic (PLEG): container finished" podID="56375571-cada-4ce6-a1b7-54be99ab9904" containerID="c630b1859c5975fab709080036f20cb3411377757e911533277a5d5f5af356a1" exitCode=0 Nov 02 13:59:55.636751 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:59:55.636323 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" event={"ID":"56375571-cada-4ce6-a1b7-54be99ab9904","Type":"ContainerDied","Data":"c630b1859c5975fab709080036f20cb3411377757e911533277a5d5f5af356a1"} Nov 02 13:59:56.767499 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:59:56.767472 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" Nov 02 13:59:56.849740 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:59:56.849709 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56375571-cada-4ce6-a1b7-54be99ab9904-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/56375571-cada-4ce6-a1b7-54be99ab9904-archives-path\") pod \"56375571-cada-4ce6-a1b7-54be99ab9904\" (UID: \"56375571-cada-4ce6-a1b7-54be99ab9904\") " Nov 02 13:59:56.849927 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:59:56.849778 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56375571-cada-4ce6-a1b7-54be99ab9904-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56375571-cada-4ce6-a1b7-54be99ab9904-trusted-ca-bundle\") pod \"56375571-cada-4ce6-a1b7-54be99ab9904\" (UID: \"56375571-cada-4ce6-a1b7-54be99ab9904\") " Nov 02 13:59:56.850106 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:59:56.850081 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56375571-cada-4ce6-a1b7-54be99ab9904-archives-path" pod "56375571-cada-4ce6-a1b7-54be99ab9904" (UID: "56375571-cada-4ce6-a1b7-54be99ab9904"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:59:56.850227 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:59:56.850202 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56375571-cada-4ce6-a1b7-54be99ab9904-trusted-ca-bundle" pod "56375571-cada-4ce6-a1b7-54be99ab9904" (UID: "56375571-cada-4ce6-a1b7-54be99ab9904"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:59:56.950238 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:59:56.950205 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56375571-cada-4ce6-a1b7-54be99ab9904-kube-api-access-7wrmc\" (UniqueName: \"kubernetes.io/projected/56375571-cada-4ce6-a1b7-54be99ab9904-kube-api-access-7wrmc\") pod \"56375571-cada-4ce6-a1b7-54be99ab9904\" (UID: \"56375571-cada-4ce6-a1b7-54be99ab9904\") " Nov 02 13:59:56.950426 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:59:56.950249 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56375571-cada-4ce6-a1b7-54be99ab9904-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56375571-cada-4ce6-a1b7-54be99ab9904-service-ca-bundle\") pod \"56375571-cada-4ce6-a1b7-54be99ab9904\" (UID: \"56375571-cada-4ce6-a1b7-54be99ab9904\") " Nov 02 13:59:56.950491 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:59:56.950442 4866 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56375571-cada-4ce6-a1b7-54be99ab9904-trusted-ca-bundle\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:59:56.950491 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:59:56.950461 4866 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/56375571-cada-4ce6-a1b7-54be99ab9904-archives-path\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:59:56.950682 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:59:56.950654 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56375571-cada-4ce6-a1b7-54be99ab9904-service-ca-bundle" pod "56375571-cada-4ce6-a1b7-54be99ab9904" (UID: "56375571-cada-4ce6-a1b7-54be99ab9904"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:59:56.952464 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:59:56.952445 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56375571-cada-4ce6-a1b7-54be99ab9904-kube-api-access-7wrmc" pod "56375571-cada-4ce6-a1b7-54be99ab9904" (UID: "56375571-cada-4ce6-a1b7-54be99ab9904"). InnerVolumeSpecName "kube-api-access-7wrmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:59:57.050778 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:59:57.050746 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wrmc\" (UniqueName: \"kubernetes.io/projected/56375571-cada-4ce6-a1b7-54be99ab9904-kube-api-access-7wrmc\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:59:57.050778 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:59:57.050772 4866 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56375571-cada-4ce6-a1b7-54be99ab9904-service-ca-bundle\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 13:59:57.644849 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:59:57.644823 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" Nov 02 13:59:57.645024 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:59:57.644818 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bpfbx-bmlmm" event={"ID":"56375571-cada-4ce6-a1b7-54be99ab9904","Type":"ContainerDied","Data":"43c1d03e71239afd9b77c801f6ef1c8b3d07735f90a9950d28206a2aa860c11e"} Nov 02 13:59:57.645024 ip-10-0-134-99 kubenswrapper[4866]: I1102 13:59:57.644966 4866 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43c1d03e71239afd9b77c801f6ef1c8b3d07735f90a9950d28206a2aa860c11e" Nov 02 14:00:06.007606 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:00:06.007565 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:00:36.008495 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:00:36.008449 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:01:06.008433 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:01:06.008357 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:01:36.008884 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:01:36.008814 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:02:06.008920 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:02:06.008876 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:02:14.949772 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:14.949741 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:02:14.950401 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:14.950386 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:02:31.006675 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.006460 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh"] Nov 02 14:02:31.007175 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.007154 4866 cpu_manager.go:414] "containerMap: removing container" podUID="56375571-cada-4ce6-a1b7-54be99ab9904" containerName="insights-gathering" Nov 02 14:02:31.007247 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.007180 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56375571-cada-4ce6-a1b7-54be99ab9904" containerName="insights-gathering" Nov 02 14:02:31.007328 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.007314 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="56375571-cada-4ce6-a1b7-54be99ab9904" containerName="insights-gathering" Nov 02 14:02:31.011591 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.011568 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.031839 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.031813 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 14:02:31.031839 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.031814 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 14:02:31.032053 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.031848 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 14:02:31.032053 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.031855 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 14:02:31.032123 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.032082 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xcrsw\"" Nov 02 14:02:31.042076 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.042044 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh"] Nov 02 14:02:31.102457 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.102421 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.102457 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.102461 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.102667 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.102486 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.102667 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.102573 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.102667 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.102611 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.102762 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.102678 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1c0506-8519-4820-a5ac-ecbe70a21d22-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.102794 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.102755 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnxph\" (UniqueName: \"kubernetes.io/projected/0c1c0506-8519-4820-a5ac-ecbe70a21d22-kube-api-access-qnxph\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.203768 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.203720 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.204016 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.203779 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.204016 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.203809 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.204016 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.203841 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.204016 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.203871 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1c0506-8519-4820-a5ac-ecbe70a21d22-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.204016 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.203893 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnxph\" (UniqueName: \"kubernetes.io/projected/0c1c0506-8519-4820-a5ac-ecbe70a21d22-kube-api-access-qnxph\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.204016 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.203980 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.204586 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.204560 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.206469 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.206435 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1c0506-8519-4820-a5ac-ecbe70a21d22-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.206584 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.206570 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.206712 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.206691 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.206770 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.206691 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.228453 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.228423 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnxph\" (UniqueName: \"kubernetes.io/projected/0c1c0506-8519-4820-a5ac-ecbe70a21d22-kube-api-access-qnxph\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.231347 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.231322 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-t8szh\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.322390 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.322303 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:31.636030 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.636006 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh"] Nov 02 14:02:31.638443 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:02:31.638406 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1c0506_8519_4820_a5ac_ecbe70a21d22.slice/crio-1db286366eb555e6d471735294963471d78e7d490eddfdd18a7672aac04bc4b3 WatchSource:0}: Error finding container 1db286366eb555e6d471735294963471d78e7d490eddfdd18a7672aac04bc4b3: Status 404 returned error can't find the container with id 1db286366eb555e6d471735294963471d78e7d490eddfdd18a7672aac04bc4b3 Nov 02 14:02:31.640319 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:31.640304 4866 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:02:32.184402 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:32.184365 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" event={"ID":"0c1c0506-8519-4820-a5ac-ecbe70a21d22","Type":"ContainerStarted","Data":"1db286366eb555e6d471735294963471d78e7d490eddfdd18a7672aac04bc4b3"} Nov 02 14:02:34.175577 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:34.175547 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:02:34.175870 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:34.175609 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:02:35.197589 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:35.197549 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" event={"ID":"0c1c0506-8519-4820-a5ac-ecbe70a21d22","Type":"ContainerStarted","Data":"c9cfd2d3c6784425cf5e64d68e75b8cf4691e1fa592b08d9bafcc277bf12eeec"} Nov 02 14:02:35.198002 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:35.197852 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:35.199230 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:35.199209 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:02:35.274762 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:02:35.274700 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" podStartSLOduration=2.740382543 podStartE2EDuration="5.274684188s" totalImagesPullingTime="2.534301645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:35.262117043 +0000 UTC m=+320.445964391" watchObservedRunningTime="2026-11-02 14:02:35.274684188 +0000 UTC m=+320.458531516" Nov 02 14:02:36.009306 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:02:36.009259 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:03:06.007692 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:03:06.007653 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:03:16.242192 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.242156 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd"] Nov 02 14:03:16.250436 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.250414 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.296425 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.296392 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd"] Nov 02 14:03:16.326818 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.326782 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 14:03:16.326818 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.326796 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 14:03:16.327046 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.326868 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 14:03:16.327046 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.327015 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 14:03:16.327128 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.327087 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 14:03:16.327128 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.327095 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-sd8v9\"" Nov 02 14:03:16.367189 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.367155 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2572f605-cbc3-4f28-b54a-ae8b597dbdfc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-dnwzd\" (UID: \"2572f605-cbc3-4f28-b54a-ae8b597dbdfc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.367189 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.367192 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2572f605-cbc3-4f28-b54a-ae8b597dbdfc-manifests\") pod \"opendatahub-operator-controller-manager-78cdb466dc-dnwzd\" (UID: \"2572f605-cbc3-4f28-b54a-ae8b597dbdfc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.367386 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.367213 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2572f605-cbc3-4f28-b54a-ae8b597dbdfc-metrics-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-dnwzd\" (UID: \"2572f605-cbc3-4f28-b54a-ae8b597dbdfc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.367386 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.367282 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2572f605-cbc3-4f28-b54a-ae8b597dbdfc-webhook-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-dnwzd\" (UID: \"2572f605-cbc3-4f28-b54a-ae8b597dbdfc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.367386 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.367322 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8629\" (UniqueName: \"kubernetes.io/projected/2572f605-cbc3-4f28-b54a-ae8b597dbdfc-kube-api-access-w8629\") pod \"opendatahub-operator-controller-manager-78cdb466dc-dnwzd\" (UID: \"2572f605-cbc3-4f28-b54a-ae8b597dbdfc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.367386 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.367366 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2572f605-cbc3-4f28-b54a-ae8b597dbdfc-tmp\") pod \"opendatahub-operator-controller-manager-78cdb466dc-dnwzd\" (UID: \"2572f605-cbc3-4f28-b54a-ae8b597dbdfc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.468367 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.468332 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2572f605-cbc3-4f28-b54a-ae8b597dbdfc-metrics-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-dnwzd\" (UID: \"2572f605-cbc3-4f28-b54a-ae8b597dbdfc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.468526 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.468373 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2572f605-cbc3-4f28-b54a-ae8b597dbdfc-webhook-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-dnwzd\" (UID: \"2572f605-cbc3-4f28-b54a-ae8b597dbdfc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.468526 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.468472 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8629\" (UniqueName: \"kubernetes.io/projected/2572f605-cbc3-4f28-b54a-ae8b597dbdfc-kube-api-access-w8629\") pod \"opendatahub-operator-controller-manager-78cdb466dc-dnwzd\" (UID: \"2572f605-cbc3-4f28-b54a-ae8b597dbdfc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.468526 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.468516 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2572f605-cbc3-4f28-b54a-ae8b597dbdfc-tmp\") pod \"opendatahub-operator-controller-manager-78cdb466dc-dnwzd\" (UID: \"2572f605-cbc3-4f28-b54a-ae8b597dbdfc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.468672 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.468547 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2572f605-cbc3-4f28-b54a-ae8b597dbdfc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-dnwzd\" (UID: \"2572f605-cbc3-4f28-b54a-ae8b597dbdfc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.468672 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.468576 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2572f605-cbc3-4f28-b54a-ae8b597dbdfc-manifests\") pod \"opendatahub-operator-controller-manager-78cdb466dc-dnwzd\" (UID: \"2572f605-cbc3-4f28-b54a-ae8b597dbdfc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.469051 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.469023 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2572f605-cbc3-4f28-b54a-ae8b597dbdfc-manifests\") pod \"opendatahub-operator-controller-manager-78cdb466dc-dnwzd\" (UID: \"2572f605-cbc3-4f28-b54a-ae8b597dbdfc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.471057 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.471033 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2572f605-cbc3-4f28-b54a-ae8b597dbdfc-webhook-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-dnwzd\" (UID: \"2572f605-cbc3-4f28-b54a-ae8b597dbdfc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.471158 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.471103 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2572f605-cbc3-4f28-b54a-ae8b597dbdfc-metrics-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-dnwzd\" (UID: \"2572f605-cbc3-4f28-b54a-ae8b597dbdfc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.471237 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.471218 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2572f605-cbc3-4f28-b54a-ae8b597dbdfc-tmp\") pod \"opendatahub-operator-controller-manager-78cdb466dc-dnwzd\" (UID: \"2572f605-cbc3-4f28-b54a-ae8b597dbdfc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.471361 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.471341 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2572f605-cbc3-4f28-b54a-ae8b597dbdfc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-dnwzd\" (UID: \"2572f605-cbc3-4f28-b54a-ae8b597dbdfc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.493372 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.493320 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8629\" (UniqueName: \"kubernetes.io/projected/2572f605-cbc3-4f28-b54a-ae8b597dbdfc-kube-api-access-w8629\") pod \"opendatahub-operator-controller-manager-78cdb466dc-dnwzd\" (UID: \"2572f605-cbc3-4f28-b54a-ae8b597dbdfc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.565296 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.565259 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:16.917992 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:16.917969 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd"] Nov 02 14:03:16.920322 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:03:16.920297 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2572f605_cbc3_4f28_b54a_ae8b597dbdfc.slice/crio-d1224b810b37470fbd658ab1fe036f9c13db43f570ab4386904cf0c312402382 WatchSource:0}: Error finding container d1224b810b37470fbd658ab1fe036f9c13db43f570ab4386904cf0c312402382: Status 404 returned error can't find the container with id d1224b810b37470fbd658ab1fe036f9c13db43f570ab4386904cf0c312402382 Nov 02 14:03:17.353715 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:17.353671 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" event={"ID":"2572f605-cbc3-4f28-b54a-ae8b597dbdfc","Type":"ContainerStarted","Data":"d1224b810b37470fbd658ab1fe036f9c13db43f570ab4386904cf0c312402382"} Nov 02 14:03:20.367125 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:20.367090 4866 generic.go:356] "Generic (PLEG): container finished" podID="2572f605-cbc3-4f28-b54a-ae8b597dbdfc" containerID="2d42dba9f9944d0fd3ca30e7bade395ff38c6b29dce2bb260de899f65ca787b3" exitCode=0 Nov 02 14:03:20.367125 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:20.367128 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" event={"ID":"2572f605-cbc3-4f28-b54a-ae8b597dbdfc","Type":"ContainerDied","Data":"2d42dba9f9944d0fd3ca30e7bade395ff38c6b29dce2bb260de899f65ca787b3"} Nov 02 14:03:21.372620 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:21.372588 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" event={"ID":"2572f605-cbc3-4f28-b54a-ae8b597dbdfc","Type":"ContainerStarted","Data":"5fe1a34a523b0154b68da621a836c1789cb2fa3de2f54bb0e47a6d84f31b6846"} Nov 02 14:03:21.373041 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:21.372717 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:21.523916 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:21.523859 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" podStartSLOduration=1.904558093 podStartE2EDuration="5.523845352s" totalImagesPullingTime="3.15637844s" totalInitContainerRuntime="462.908819ms" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:21.511700748 +0000 UTC m=+366.695548088" watchObservedRunningTime="2026-11-02 14:03:21.523845352 +0000 UTC m=+366.707692688" Nov 02 14:03:32.379122 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:32.379088 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-dnwzd" Nov 02 14:03:36.008259 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:03:36.008223 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:03:53.258692 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:53.258655 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8f9c48f5f-jlflv"] Nov 02 14:03:53.321844 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:53.321811 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8f9c48f5f-jlflv"] Nov 02 14:03:53.322037 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:53.321855 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-jlflv" Nov 02 14:03:53.352476 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:53.352443 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67f443f5-5750-4236-bcf7-b3239b253c9e-tmp\") pod \"kube-auth-proxy-8f9c48f5f-jlflv\" (UID: \"67f443f5-5750-4236-bcf7-b3239b253c9e\") " pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-jlflv" Nov 02 14:03:53.352636 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:53.352483 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f443f5-5750-4236-bcf7-b3239b253c9e-tls-certs\") pod \"kube-auth-proxy-8f9c48f5f-jlflv\" (UID: \"67f443f5-5750-4236-bcf7-b3239b253c9e\") " pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-jlflv" Nov 02 14:03:53.352636 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:53.352620 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5cvz\" (UniqueName: \"kubernetes.io/projected/67f443f5-5750-4236-bcf7-b3239b253c9e-kube-api-access-q5cvz\") pod \"kube-auth-proxy-8f9c48f5f-jlflv\" (UID: \"67f443f5-5750-4236-bcf7-b3239b253c9e\") " pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-jlflv" Nov 02 14:03:53.394582 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:53.394546 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 14:03:53.394582 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:53.394546 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 14:03:53.394855 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:53.394838 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-j28bz\"" Nov 02 14:03:53.453766 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:53.453735 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f443f5-5750-4236-bcf7-b3239b253c9e-tls-certs\") pod \"kube-auth-proxy-8f9c48f5f-jlflv\" (UID: \"67f443f5-5750-4236-bcf7-b3239b253c9e\") " pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-jlflv" Nov 02 14:03:53.453918 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:53.453799 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5cvz\" (UniqueName: \"kubernetes.io/projected/67f443f5-5750-4236-bcf7-b3239b253c9e-kube-api-access-q5cvz\") pod \"kube-auth-proxy-8f9c48f5f-jlflv\" (UID: \"67f443f5-5750-4236-bcf7-b3239b253c9e\") " pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-jlflv" Nov 02 14:03:53.453918 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:53.453842 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67f443f5-5750-4236-bcf7-b3239b253c9e-tmp\") pod \"kube-auth-proxy-8f9c48f5f-jlflv\" (UID: \"67f443f5-5750-4236-bcf7-b3239b253c9e\") " pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-jlflv" Nov 02 14:03:53.456295 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:53.456218 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67f443f5-5750-4236-bcf7-b3239b253c9e-tmp\") pod \"kube-auth-proxy-8f9c48f5f-jlflv\" (UID: \"67f443f5-5750-4236-bcf7-b3239b253c9e\") " pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-jlflv" Nov 02 14:03:53.456492 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:53.456472 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f443f5-5750-4236-bcf7-b3239b253c9e-tls-certs\") pod \"kube-auth-proxy-8f9c48f5f-jlflv\" (UID: \"67f443f5-5750-4236-bcf7-b3239b253c9e\") " pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-jlflv" Nov 02 14:03:53.485664 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:53.485636 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5cvz\" (UniqueName: \"kubernetes.io/projected/67f443f5-5750-4236-bcf7-b3239b253c9e-kube-api-access-q5cvz\") pod \"kube-auth-proxy-8f9c48f5f-jlflv\" (UID: \"67f443f5-5750-4236-bcf7-b3239b253c9e\") " pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-jlflv" Nov 02 14:03:53.647059 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:53.646972 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-jlflv" Nov 02 14:03:53.964215 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:53.964104 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8f9c48f5f-jlflv"] Nov 02 14:03:53.966808 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:03:53.966780 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f443f5_5750_4236_bcf7_b3239b253c9e.slice/crio-deb8da2dc728e8352e273598a9570a4d117ac1ffa33e3035ccd7c38720893f79 WatchSource:0}: Error finding container deb8da2dc728e8352e273598a9570a4d117ac1ffa33e3035ccd7c38720893f79: Status 404 returned error can't find the container with id deb8da2dc728e8352e273598a9570a4d117ac1ffa33e3035ccd7c38720893f79 Nov 02 14:03:54.504936 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:54.504890 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-jlflv" event={"ID":"67f443f5-5750-4236-bcf7-b3239b253c9e","Type":"ContainerStarted","Data":"deb8da2dc728e8352e273598a9570a4d117ac1ffa33e3035ccd7c38720893f79"} Nov 02 14:03:56.515479 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:56.515447 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-jlflv" event={"ID":"67f443f5-5750-4236-bcf7-b3239b253c9e","Type":"ContainerStarted","Data":"35c7bbe1d6457f5b4efb5ed0ea405c33ed03020143d9ea2d49a7141b21a188ec"} Nov 02 14:03:56.557782 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:03:56.557681 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-jlflv" podStartSLOduration=1.2446441560000001 podStartE2EDuration="3.557667828s" totalImagesPullingTime="2.313023672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:56.556608063 +0000 UTC m=+401.740455400" watchObservedRunningTime="2026-11-02 14:03:56.557667828 +0000 UTC m=+401.741515165" Nov 02 14:04:06.008994 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:04:06.008903 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:04:21.290740 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.290703 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn"] Nov 02 14:04:21.293906 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.293885 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn" Nov 02 14:04:21.325472 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.325437 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-gljs4\"" Nov 02 14:04:21.325763 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.325742 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 02 14:04:21.325866 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.325827 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 02 14:04:21.325924 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.325892 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 02 14:04:21.326063 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.326047 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 14:04:21.327588 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.327569 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn"] Nov 02 14:04:21.384140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.384109 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/778bb275-ee0e-4a6f-88a5-774324a3d353-metrics-certs\") pod \"jobset-controller-manager-67b8558cc4-5qlrn\" (UID: \"778bb275-ee0e-4a6f-88a5-774324a3d353\") " pod="openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn" Nov 02 14:04:21.384293 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.384176 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/778bb275-ee0e-4a6f-88a5-774324a3d353-cert\") pod \"jobset-controller-manager-67b8558cc4-5qlrn\" (UID: \"778bb275-ee0e-4a6f-88a5-774324a3d353\") " pod="openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn" Nov 02 14:04:21.384293 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.384243 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwsg2\" (UniqueName: \"kubernetes.io/projected/778bb275-ee0e-4a6f-88a5-774324a3d353-kube-api-access-vwsg2\") pod \"jobset-controller-manager-67b8558cc4-5qlrn\" (UID: \"778bb275-ee0e-4a6f-88a5-774324a3d353\") " pod="openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn" Nov 02 14:04:21.384293 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.384270 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/778bb275-ee0e-4a6f-88a5-774324a3d353-manager-config\") pod \"jobset-controller-manager-67b8558cc4-5qlrn\" (UID: \"778bb275-ee0e-4a6f-88a5-774324a3d353\") " pod="openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn" Nov 02 14:04:21.413405 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.413374 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:04:21.484938 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.484899 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/778bb275-ee0e-4a6f-88a5-774324a3d353-cert\") pod \"jobset-controller-manager-67b8558cc4-5qlrn\" (UID: \"778bb275-ee0e-4a6f-88a5-774324a3d353\") " pod="openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn" Nov 02 14:04:21.485126 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.484968 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwsg2\" (UniqueName: \"kubernetes.io/projected/778bb275-ee0e-4a6f-88a5-774324a3d353-kube-api-access-vwsg2\") pod \"jobset-controller-manager-67b8558cc4-5qlrn\" (UID: \"778bb275-ee0e-4a6f-88a5-774324a3d353\") " pod="openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn" Nov 02 14:04:21.485126 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.484999 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/778bb275-ee0e-4a6f-88a5-774324a3d353-manager-config\") pod \"jobset-controller-manager-67b8558cc4-5qlrn\" (UID: \"778bb275-ee0e-4a6f-88a5-774324a3d353\") " pod="openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn" Nov 02 14:04:21.485126 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.485032 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/778bb275-ee0e-4a6f-88a5-774324a3d353-metrics-certs\") pod \"jobset-controller-manager-67b8558cc4-5qlrn\" (UID: \"778bb275-ee0e-4a6f-88a5-774324a3d353\") " pod="openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn" Nov 02 14:04:21.485685 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.485660 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/778bb275-ee0e-4a6f-88a5-774324a3d353-manager-config\") pod \"jobset-controller-manager-67b8558cc4-5qlrn\" (UID: \"778bb275-ee0e-4a6f-88a5-774324a3d353\") " pod="openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn" Nov 02 14:04:21.487491 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.487469 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/778bb275-ee0e-4a6f-88a5-774324a3d353-metrics-certs\") pod \"jobset-controller-manager-67b8558cc4-5qlrn\" (UID: \"778bb275-ee0e-4a6f-88a5-774324a3d353\") " pod="openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn" Nov 02 14:04:21.487573 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.487519 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/778bb275-ee0e-4a6f-88a5-774324a3d353-cert\") pod \"jobset-controller-manager-67b8558cc4-5qlrn\" (UID: \"778bb275-ee0e-4a6f-88a5-774324a3d353\") " pod="openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn" Nov 02 14:04:21.509592 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.509562 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwsg2\" (UniqueName: \"kubernetes.io/projected/778bb275-ee0e-4a6f-88a5-774324a3d353-kube-api-access-vwsg2\") pod \"jobset-controller-manager-67b8558cc4-5qlrn\" (UID: \"778bb275-ee0e-4a6f-88a5-774324a3d353\") " pod="openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn" Nov 02 14:04:21.607968 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.607865 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn" Nov 02 14:04:21.912179 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:21.912101 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn"] Nov 02 14:04:21.915344 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:04:21.915322 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778bb275_ee0e_4a6f_88a5_774324a3d353.slice/crio-eef844cf5d8dc6423103c5160ff50911bcb45b4cc9bd389bb73d7844a603409f WatchSource:0}: Error finding container eef844cf5d8dc6423103c5160ff50911bcb45b4cc9bd389bb73d7844a603409f: Status 404 returned error can't find the container with id eef844cf5d8dc6423103c5160ff50911bcb45b4cc9bd389bb73d7844a603409f Nov 02 14:04:22.618169 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:22.618137 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn" event={"ID":"778bb275-ee0e-4a6f-88a5-774324a3d353","Type":"ContainerStarted","Data":"eef844cf5d8dc6423103c5160ff50911bcb45b4cc9bd389bb73d7844a603409f"} Nov 02 14:04:25.632806 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:25.632765 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn" event={"ID":"778bb275-ee0e-4a6f-88a5-774324a3d353","Type":"ContainerStarted","Data":"da59ee707a448ee611a80b924f2ac0fe636aeac4ef6a9117eea86ff331399c79"} Nov 02 14:04:25.633273 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:25.632884 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn" Nov 02 14:04:25.684181 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:25.684131 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn" podStartSLOduration=1.9671387299999998 podStartE2EDuration="4.684118698s" totalImagesPullingTime="2.716979968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:04:25.682776342 +0000 UTC m=+430.866623675" watchObservedRunningTime="2026-11-02 14:04:25.684118698 +0000 UTC m=+430.867966035" Nov 02 14:04:36.009685 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:04:36.009647 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:04:36.642670 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:04:36.642633 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-67b8558cc4-5qlrn" Nov 02 14:05:04.054984 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.054933 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh"] Nov 02 14:05:04.055853 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.055185 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" podUID="0c1c0506-8519-4820-a5ac-ecbe70a21d22" containerName="discovery" containerID="cri-o://c9cfd2d3c6784425cf5e64d68e75b8cf4691e1fa592b08d9bafcc277bf12eeec" gracePeriod=30 Nov 02 14:05:04.300900 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.300875 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:05:04.433902 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.433760 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-csr-ca-configmap\") pod \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " Nov 02 14:05:04.434083 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.433915 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-token\" (UniqueName: \"kubernetes.io/projected/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-token\") pod \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " Nov 02 14:05:04.434083 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.433975 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-kubeconfig\") pod \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " Nov 02 14:05:04.434083 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.434004 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-csr-dns-cert\") pod \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " Nov 02 14:05:04.434083 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.434030 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-cacerts\" (UniqueName: \"kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-cacerts\") pod \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " Nov 02 14:05:04.434083 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.434062 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c1c0506-8519-4820-a5ac-ecbe70a21d22-kube-api-access-qnxph\" (UniqueName: \"kubernetes.io/projected/0c1c0506-8519-4820-a5ac-ecbe70a21d22-kube-api-access-qnxph\") pod \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " Nov 02 14:05:04.434375 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.434124 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1c0506-8519-4820-a5ac-ecbe70a21d22-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1c0506-8519-4820-a5ac-ecbe70a21d22-local-certs\") pod \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\" (UID: \"0c1c0506-8519-4820-a5ac-ecbe70a21d22\") " Nov 02 14:05:04.434375 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.434176 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-csr-ca-configmap" pod "0c1c0506-8519-4820-a5ac-ecbe70a21d22" (UID: "0c1c0506-8519-4820-a5ac-ecbe70a21d22"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:05:04.434498 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.434380 4866 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-csr-ca-configmap\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:05:04.436384 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.436355 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-kubeconfig" pod "0c1c0506-8519-4820-a5ac-ecbe70a21d22" (UID: "0c1c0506-8519-4820-a5ac-ecbe70a21d22"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:05:04.436514 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.436381 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-csr-dns-cert" pod "0c1c0506-8519-4820-a5ac-ecbe70a21d22" (UID: "0c1c0506-8519-4820-a5ac-ecbe70a21d22"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:05:04.436621 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.436594 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1c0506-8519-4820-a5ac-ecbe70a21d22-kube-api-access-qnxph" pod "0c1c0506-8519-4820-a5ac-ecbe70a21d22" (UID: "0c1c0506-8519-4820-a5ac-ecbe70a21d22"). InnerVolumeSpecName "kube-api-access-qnxph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:05:04.436718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.436653 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-token" pod "0c1c0506-8519-4820-a5ac-ecbe70a21d22" (UID: "0c1c0506-8519-4820-a5ac-ecbe70a21d22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:05:04.436718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.436650 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-cacerts" pod "0c1c0506-8519-4820-a5ac-ecbe70a21d22" (UID: "0c1c0506-8519-4820-a5ac-ecbe70a21d22"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:05:04.436718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.436685 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1c0506-8519-4820-a5ac-ecbe70a21d22-local-certs" pod "0c1c0506-8519-4820-a5ac-ecbe70a21d22" (UID: "0c1c0506-8519-4820-a5ac-ecbe70a21d22"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:05:04.535267 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.535227 4866 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-csr-dns-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:05:04.535267 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.535262 4866 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-cacerts\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:05:04.535267 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.535274 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnxph\" (UniqueName: \"kubernetes.io/projected/0c1c0506-8519-4820-a5ac-ecbe70a21d22-kube-api-access-qnxph\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:05:04.535490 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.535284 4866 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1c0506-8519-4820-a5ac-ecbe70a21d22-local-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:05:04.535490 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.535293 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:05:04.535490 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.535301 4866 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c1c0506-8519-4820-a5ac-ecbe70a21d22-istio-kubeconfig\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:05:04.787125 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.787089 4866 generic.go:356] "Generic (PLEG): container finished" podID="0c1c0506-8519-4820-a5ac-ecbe70a21d22" containerID="c9cfd2d3c6784425cf5e64d68e75b8cf4691e1fa592b08d9bafcc277bf12eeec" exitCode=0 Nov 02 14:05:04.787273 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.787140 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" Nov 02 14:05:04.787273 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.787171 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" event={"ID":"0c1c0506-8519-4820-a5ac-ecbe70a21d22","Type":"ContainerDied","Data":"c9cfd2d3c6784425cf5e64d68e75b8cf4691e1fa592b08d9bafcc277bf12eeec"} Nov 02 14:05:04.787273 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.787222 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh" event={"ID":"0c1c0506-8519-4820-a5ac-ecbe70a21d22","Type":"ContainerDied","Data":"1db286366eb555e6d471735294963471d78e7d490eddfdd18a7672aac04bc4b3"} Nov 02 14:05:04.787273 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.787242 4866 scope.go:122] "RemoveContainer" containerID="c9cfd2d3c6784425cf5e64d68e75b8cf4691e1fa592b08d9bafcc277bf12eeec" Nov 02 14:05:04.795497 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.795471 4866 scope.go:122] "RemoveContainer" containerID="c9cfd2d3c6784425cf5e64d68e75b8cf4691e1fa592b08d9bafcc277bf12eeec" Nov 02 14:05:04.795787 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:05:04.795766 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9cfd2d3c6784425cf5e64d68e75b8cf4691e1fa592b08d9bafcc277bf12eeec\": container with ID starting with c9cfd2d3c6784425cf5e64d68e75b8cf4691e1fa592b08d9bafcc277bf12eeec not found: ID does not exist" containerID="c9cfd2d3c6784425cf5e64d68e75b8cf4691e1fa592b08d9bafcc277bf12eeec" Nov 02 14:05:04.795853 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.795796 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9cfd2d3c6784425cf5e64d68e75b8cf4691e1fa592b08d9bafcc277bf12eeec"} err="failed to get container status \"c9cfd2d3c6784425cf5e64d68e75b8cf4691e1fa592b08d9bafcc277bf12eeec\": rpc error: code = NotFound desc = could not find container \"c9cfd2d3c6784425cf5e64d68e75b8cf4691e1fa592b08d9bafcc277bf12eeec\": container with ID starting with c9cfd2d3c6784425cf5e64d68e75b8cf4691e1fa592b08d9bafcc277bf12eeec not found: ID does not exist" Nov 02 14:05:04.851309 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.851274 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh"] Nov 02 14:05:04.863469 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.863438 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-t8szh"] Nov 02 14:05:04.948603 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:04.948573 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c1c0506-8519-4820-a5ac-ecbe70a21d22" path="/var/lib/kubelet/pods/0c1c0506-8519-4820-a5ac-ecbe70a21d22/volumes" Nov 02 14:05:06.008518 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:05:06.008486 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:05:08.349734 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.349695 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh"] Nov 02 14:05:08.350188 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.350022 4866 cpu_manager.go:414] "containerMap: removing container" podUID="0c1c0506-8519-4820-a5ac-ecbe70a21d22" containerName="discovery" Nov 02 14:05:08.350188 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.350034 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1c0506-8519-4820-a5ac-ecbe70a21d22" containerName="discovery" Nov 02 14:05:08.350188 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.350087 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1c0506-8519-4820-a5ac-ecbe70a21d22" containerName="discovery" Nov 02 14:05:08.353121 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.353101 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh" Nov 02 14:05:08.368257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.368237 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 02 14:05:08.379984 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.379937 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 02 14:05:08.379984 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.379974 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 14:05:08.380174 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.380011 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 02 14:05:08.380174 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.380035 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-9xcr5\"" Nov 02 14:05:08.380174 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.380105 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 14:05:08.381981 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.381935 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh"] Nov 02 14:05:08.470905 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.470867 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36201127-d7ae-4a96-879f-d28da880063c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c2cbh\" (UID: \"36201127-d7ae-4a96-879f-d28da880063c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh" Nov 02 14:05:08.471085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.470915 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/36201127-d7ae-4a96-879f-d28da880063c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c2cbh\" (UID: \"36201127-d7ae-4a96-879f-d28da880063c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh" Nov 02 14:05:08.471085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.471045 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36201127-d7ae-4a96-879f-d28da880063c-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c2cbh\" (UID: \"36201127-d7ae-4a96-879f-d28da880063c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh" Nov 02 14:05:08.471085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.471075 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zpk8\" (UniqueName: \"kubernetes.io/projected/36201127-d7ae-4a96-879f-d28da880063c-kube-api-access-6zpk8\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c2cbh\" (UID: \"36201127-d7ae-4a96-879f-d28da880063c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh" Nov 02 14:05:08.571597 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.571558 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/36201127-d7ae-4a96-879f-d28da880063c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c2cbh\" (UID: \"36201127-d7ae-4a96-879f-d28da880063c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh" Nov 02 14:05:08.571767 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.571614 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36201127-d7ae-4a96-879f-d28da880063c-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c2cbh\" (UID: \"36201127-d7ae-4a96-879f-d28da880063c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh" Nov 02 14:05:08.571767 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.571640 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zpk8\" (UniqueName: \"kubernetes.io/projected/36201127-d7ae-4a96-879f-d28da880063c-kube-api-access-6zpk8\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c2cbh\" (UID: \"36201127-d7ae-4a96-879f-d28da880063c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh" Nov 02 14:05:08.571767 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.571673 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36201127-d7ae-4a96-879f-d28da880063c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c2cbh\" (UID: \"36201127-d7ae-4a96-879f-d28da880063c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh" Nov 02 14:05:08.572375 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.572346 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36201127-d7ae-4a96-879f-d28da880063c-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c2cbh\" (UID: \"36201127-d7ae-4a96-879f-d28da880063c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh" Nov 02 14:05:08.572493 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.572430 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/36201127-d7ae-4a96-879f-d28da880063c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c2cbh\" (UID: \"36201127-d7ae-4a96-879f-d28da880063c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh" Nov 02 14:05:08.574069 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.574050 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36201127-d7ae-4a96-879f-d28da880063c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c2cbh\" (UID: \"36201127-d7ae-4a96-879f-d28da880063c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh" Nov 02 14:05:08.595170 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.595141 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zpk8\" (UniqueName: \"kubernetes.io/projected/36201127-d7ae-4a96-879f-d28da880063c-kube-api-access-6zpk8\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c2cbh\" (UID: \"36201127-d7ae-4a96-879f-d28da880063c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh" Nov 02 14:05:08.664583 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.664508 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh" Nov 02 14:05:08.960467 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:08.960439 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh"] Nov 02 14:05:08.964112 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:05:08.964081 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36201127_d7ae_4a96_879f_d28da880063c.slice/crio-c82d59246505acda6bcbd42bf60ecd52d2b8c7d2692c1cfaabd7b53c86e92c84 WatchSource:0}: Error finding container c82d59246505acda6bcbd42bf60ecd52d2b8c7d2692c1cfaabd7b53c86e92c84: Status 404 returned error can't find the container with id c82d59246505acda6bcbd42bf60ecd52d2b8c7d2692c1cfaabd7b53c86e92c84 Nov 02 14:05:09.809193 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:09.809158 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh" event={"ID":"36201127-d7ae-4a96-879f-d28da880063c","Type":"ContainerStarted","Data":"c82d59246505acda6bcbd42bf60ecd52d2b8c7d2692c1cfaabd7b53c86e92c84"} Nov 02 14:05:19.857453 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:19.857419 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh" event={"ID":"36201127-d7ae-4a96-879f-d28da880063c","Type":"ContainerStarted","Data":"a14cf08c21df42f9da19fc055350cdb5689e815e529382557fe4499f310a807c"} Nov 02 14:05:19.857848 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:19.857551 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh" Nov 02 14:05:19.899811 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:19.899758 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh" podStartSLOduration=1.529087095 podStartE2EDuration="11.899745196s" totalImagesPullingTime="10.370658101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:19.898979689 +0000 UTC m=+485.082827025" watchObservedRunningTime="2026-11-02 14:05:19.899745196 +0000 UTC m=+485.083592534" Nov 02 14:05:30.867668 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:30.867593 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c2cbh" Nov 02 14:05:36.008744 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:05:36.008701 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:05:39.593189 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.593152 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-54889d7857-j65vr"] Nov 02 14:05:39.598645 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.598622 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" Nov 02 14:05:39.614176 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.614152 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 14:05:39.626097 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.626070 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 14:05:39.626285 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.626080 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-245cc\"" Nov 02 14:05:39.626285 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.626080 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 14:05:39.626416 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.626169 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 14:05:39.627499 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.627479 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54889d7857-j65vr"] Nov 02 14:05:39.646933 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.646904 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a600929e-ff62-427a-89dc-9f1eb7f71256-openshift-service-ca\") pod \"maas-api-54889d7857-j65vr\" (UID: \"a600929e-ff62-427a-89dc-9f1eb7f71256\") " pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" Nov 02 14:05:39.647111 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.646962 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h845\" (UniqueName: \"kubernetes.io/projected/a600929e-ff62-427a-89dc-9f1eb7f71256-kube-api-access-6h845\") pod \"maas-api-54889d7857-j65vr\" (UID: \"a600929e-ff62-427a-89dc-9f1eb7f71256\") " pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" Nov 02 14:05:39.647111 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.647104 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a600929e-ff62-427a-89dc-9f1eb7f71256-maas-api-tls\") pod \"maas-api-54889d7857-j65vr\" (UID: \"a600929e-ff62-427a-89dc-9f1eb7f71256\") " pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" Nov 02 14:05:39.647223 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.647130 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a600929e-ff62-427a-89dc-9f1eb7f71256-metrics-certs\") pod \"maas-api-54889d7857-j65vr\" (UID: \"a600929e-ff62-427a-89dc-9f1eb7f71256\") " pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" Nov 02 14:05:39.748164 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.748124 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a600929e-ff62-427a-89dc-9f1eb7f71256-maas-api-tls\") pod \"maas-api-54889d7857-j65vr\" (UID: \"a600929e-ff62-427a-89dc-9f1eb7f71256\") " pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" Nov 02 14:05:39.748164 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.748162 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a600929e-ff62-427a-89dc-9f1eb7f71256-metrics-certs\") pod \"maas-api-54889d7857-j65vr\" (UID: \"a600929e-ff62-427a-89dc-9f1eb7f71256\") " pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" Nov 02 14:05:39.748377 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.748197 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a600929e-ff62-427a-89dc-9f1eb7f71256-openshift-service-ca\") pod \"maas-api-54889d7857-j65vr\" (UID: \"a600929e-ff62-427a-89dc-9f1eb7f71256\") " pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" Nov 02 14:05:39.748377 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.748226 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h845\" (UniqueName: \"kubernetes.io/projected/a600929e-ff62-427a-89dc-9f1eb7f71256-kube-api-access-6h845\") pod \"maas-api-54889d7857-j65vr\" (UID: \"a600929e-ff62-427a-89dc-9f1eb7f71256\") " pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" Nov 02 14:05:39.748910 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.748887 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a600929e-ff62-427a-89dc-9f1eb7f71256-openshift-service-ca\") pod \"maas-api-54889d7857-j65vr\" (UID: \"a600929e-ff62-427a-89dc-9f1eb7f71256\") " pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" Nov 02 14:05:39.750720 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.750696 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a600929e-ff62-427a-89dc-9f1eb7f71256-metrics-certs\") pod \"maas-api-54889d7857-j65vr\" (UID: \"a600929e-ff62-427a-89dc-9f1eb7f71256\") " pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" Nov 02 14:05:39.750813 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.750743 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a600929e-ff62-427a-89dc-9f1eb7f71256-maas-api-tls\") pod \"maas-api-54889d7857-j65vr\" (UID: \"a600929e-ff62-427a-89dc-9f1eb7f71256\") " pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" Nov 02 14:05:39.772213 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.772184 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h845\" (UniqueName: \"kubernetes.io/projected/a600929e-ff62-427a-89dc-9f1eb7f71256-kube-api-access-6h845\") pod \"maas-api-54889d7857-j65vr\" (UID: \"a600929e-ff62-427a-89dc-9f1eb7f71256\") " pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" Nov 02 14:05:39.910705 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:39.910615 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" Nov 02 14:05:40.208085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:40.208051 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54889d7857-j65vr"] Nov 02 14:05:40.211747 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:05:40.211715 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda600929e_ff62_427a_89dc_9f1eb7f71256.slice/crio-5dbf063ab65916fc9ad7af3bfecdaeeb6017cf2ba1d537132c2f48ef7a38fe43 WatchSource:0}: Error finding container 5dbf063ab65916fc9ad7af3bfecdaeeb6017cf2ba1d537132c2f48ef7a38fe43: Status 404 returned error can't find the container with id 5dbf063ab65916fc9ad7af3bfecdaeeb6017cf2ba1d537132c2f48ef7a38fe43 Nov 02 14:05:40.945162 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:40.945123 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" event={"ID":"a600929e-ff62-427a-89dc-9f1eb7f71256","Type":"ContainerStarted","Data":"5dbf063ab65916fc9ad7af3bfecdaeeb6017cf2ba1d537132c2f48ef7a38fe43"} Nov 02 14:05:43.958861 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:43.958820 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" event={"ID":"a600929e-ff62-427a-89dc-9f1eb7f71256","Type":"ContainerStarted","Data":"49dc5d7ea88637f80fcce1fc7af788f58640104252d3ace4231840720259b131"} Nov 02 14:05:43.959321 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:43.958934 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" Nov 02 14:05:44.003795 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:44.003741 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" podStartSLOduration=1.857163766 podStartE2EDuration="5.003727957s" totalImagesPullingTime="3.146564191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:44.002735118 +0000 UTC m=+509.186582457" watchObservedRunningTime="2026-11-02 14:05:44.003727957 +0000 UTC m=+509.187575294" Nov 02 14:05:49.967408 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:49.967377 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" Nov 02 14:05:50.800740 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:50.800702 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hv5z8"] Nov 02 14:05:50.813437 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:50.813412 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hv5z8" Nov 02 14:05:50.824684 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:50.824656 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hv5z8"] Nov 02 14:05:50.835045 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:50.835022 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhbvw\" (UniqueName: \"kubernetes.io/projected/43378270-dfc5-482c-aaec-97eb49a47481-kube-api-access-dhbvw\") pod \"authorino-6f6944bcdd-hv5z8\" (UID: \"43378270-dfc5-482c-aaec-97eb49a47481\") " pod="kuadrant-system/authorino-6f6944bcdd-hv5z8" Nov 02 14:05:50.835531 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:50.835513 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5hbj7\"" Nov 02 14:05:50.861500 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:50.861480 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 14:05:50.861613 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:50.861537 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 14:05:50.936432 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:50.936393 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhbvw\" (UniqueName: \"kubernetes.io/projected/43378270-dfc5-482c-aaec-97eb49a47481-kube-api-access-dhbvw\") pod \"authorino-6f6944bcdd-hv5z8\" (UID: \"43378270-dfc5-482c-aaec-97eb49a47481\") " pod="kuadrant-system/authorino-6f6944bcdd-hv5z8" Nov 02 14:05:50.962652 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:50.962622 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhbvw\" (UniqueName: \"kubernetes.io/projected/43378270-dfc5-482c-aaec-97eb49a47481-kube-api-access-dhbvw\") pod \"authorino-6f6944bcdd-hv5z8\" (UID: \"43378270-dfc5-482c-aaec-97eb49a47481\") " pod="kuadrant-system/authorino-6f6944bcdd-hv5z8" Nov 02 14:05:50.989693 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:50.989659 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr"] Nov 02 14:05:50.992145 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:50.992125 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" Nov 02 14:05:51.004936 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.004911 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hv5z8"] Nov 02 14:05:51.005162 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.005147 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hv5z8" Nov 02 14:05:51.029186 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.029158 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr"] Nov 02 14:05:51.037357 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.037327 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f38a69-4298-4c7c-a287-190d35223473-metrics-certs\") pod \"maas-api-55b5486c4c-dwrbr\" (UID: \"a1f38a69-4298-4c7c-a287-190d35223473\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" Nov 02 14:05:51.037472 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.037371 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdmh6\" (UniqueName: \"kubernetes.io/projected/a1f38a69-4298-4c7c-a287-190d35223473-kube-api-access-hdmh6\") pod \"maas-api-55b5486c4c-dwrbr\" (UID: \"a1f38a69-4298-4c7c-a287-190d35223473\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" Nov 02 14:05:51.037472 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.037459 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f38a69-4298-4c7c-a287-190d35223473-openshift-service-ca\") pod \"maas-api-55b5486c4c-dwrbr\" (UID: \"a1f38a69-4298-4c7c-a287-190d35223473\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" Nov 02 14:05:51.037545 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.037507 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1f38a69-4298-4c7c-a287-190d35223473-maas-api-tls\") pod \"maas-api-55b5486c4c-dwrbr\" (UID: \"a1f38a69-4298-4c7c-a287-190d35223473\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" Nov 02 14:05:51.050536 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.050508 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7696f"] Nov 02 14:05:51.054988 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.054972 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7696f" Nov 02 14:05:51.073456 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.073107 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 02 14:05:51.076512 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.076488 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7696f"] Nov 02 14:05:51.138300 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.138258 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4-tls-cert\") pod \"authorino-6889cbd8c4-7696f\" (UID: \"eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4\") " pod="kuadrant-system/authorino-6889cbd8c4-7696f" Nov 02 14:05:51.138483 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.138313 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8hks\" (UniqueName: \"kubernetes.io/projected/eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4-kube-api-access-k8hks\") pod \"authorino-6889cbd8c4-7696f\" (UID: \"eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4\") " pod="kuadrant-system/authorino-6889cbd8c4-7696f" Nov 02 14:05:51.138483 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.138405 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f38a69-4298-4c7c-a287-190d35223473-metrics-certs\") pod \"maas-api-55b5486c4c-dwrbr\" (UID: \"a1f38a69-4298-4c7c-a287-190d35223473\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" Nov 02 14:05:51.138483 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.138432 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmh6\" (UniqueName: \"kubernetes.io/projected/a1f38a69-4298-4c7c-a287-190d35223473-kube-api-access-hdmh6\") pod \"maas-api-55b5486c4c-dwrbr\" (UID: \"a1f38a69-4298-4c7c-a287-190d35223473\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" Nov 02 14:05:51.138483 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.138453 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f38a69-4298-4c7c-a287-190d35223473-openshift-service-ca\") pod \"maas-api-55b5486c4c-dwrbr\" (UID: \"a1f38a69-4298-4c7c-a287-190d35223473\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" Nov 02 14:05:51.138638 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.138487 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1f38a69-4298-4c7c-a287-190d35223473-maas-api-tls\") pod \"maas-api-55b5486c4c-dwrbr\" (UID: \"a1f38a69-4298-4c7c-a287-190d35223473\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" Nov 02 14:05:51.139148 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.139120 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f38a69-4298-4c7c-a287-190d35223473-openshift-service-ca\") pod \"maas-api-55b5486c4c-dwrbr\" (UID: \"a1f38a69-4298-4c7c-a287-190d35223473\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" Nov 02 14:05:51.140882 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.140861 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f38a69-4298-4c7c-a287-190d35223473-metrics-certs\") pod \"maas-api-55b5486c4c-dwrbr\" (UID: \"a1f38a69-4298-4c7c-a287-190d35223473\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" Nov 02 14:05:51.140998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.140911 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1f38a69-4298-4c7c-a287-190d35223473-maas-api-tls\") pod \"maas-api-55b5486c4c-dwrbr\" (UID: \"a1f38a69-4298-4c7c-a287-190d35223473\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" Nov 02 14:05:51.161821 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.161797 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdmh6\" (UniqueName: \"kubernetes.io/projected/a1f38a69-4298-4c7c-a287-190d35223473-kube-api-access-hdmh6\") pod \"maas-api-55b5486c4c-dwrbr\" (UID: \"a1f38a69-4298-4c7c-a287-190d35223473\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" Nov 02 14:05:51.181232 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.181205 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7696f"] Nov 02 14:05:51.181425 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:05:51.181406 4866 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-k8hks tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-7696f" podUID="eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4" Nov 02 14:05:51.217519 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.217483 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c75459655-fdx74"] Nov 02 14:05:51.220094 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.220073 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c75459655-fdx74" Nov 02 14:05:51.239133 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.239105 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8hks\" (UniqueName: \"kubernetes.io/projected/eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4-kube-api-access-k8hks\") pod \"authorino-6889cbd8c4-7696f\" (UID: \"eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4\") " pod="kuadrant-system/authorino-6889cbd8c4-7696f" Nov 02 14:05:51.239284 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.239147 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lsdb\" (UniqueName: \"kubernetes.io/projected/60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a-kube-api-access-9lsdb\") pod \"authorino-c75459655-fdx74\" (UID: \"60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a\") " pod="kuadrant-system/authorino-c75459655-fdx74" Nov 02 14:05:51.239284 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.239185 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a-tls-cert\") pod \"authorino-c75459655-fdx74\" (UID: \"60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a\") " pod="kuadrant-system/authorino-c75459655-fdx74" Nov 02 14:05:51.239366 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.239293 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4-tls-cert\") pod \"authorino-6889cbd8c4-7696f\" (UID: \"eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4\") " pod="kuadrant-system/authorino-6889cbd8c4-7696f" Nov 02 14:05:51.241874 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.241851 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4-tls-cert\") pod \"authorino-6889cbd8c4-7696f\" (UID: \"eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4\") " pod="kuadrant-system/authorino-6889cbd8c4-7696f" Nov 02 14:05:51.242510 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.242491 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c75459655-fdx74"] Nov 02 14:05:51.264229 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.264203 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8hks\" (UniqueName: \"kubernetes.io/projected/eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4-kube-api-access-k8hks\") pod \"authorino-6889cbd8c4-7696f\" (UID: \"eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4\") " pod="kuadrant-system/authorino-6889cbd8c4-7696f" Nov 02 14:05:51.303919 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.303885 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" Nov 02 14:05:51.308046 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.308017 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hv5z8"] Nov 02 14:05:51.311845 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:05:51.311822 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43378270_dfc5_482c_aaec_97eb49a47481.slice/crio-8065343c2c3ca1b83161823c5a4519dda213fa9f1b69333754a077d10e9cf08b WatchSource:0}: Error finding container 8065343c2c3ca1b83161823c5a4519dda213fa9f1b69333754a077d10e9cf08b: Status 404 returned error can't find the container with id 8065343c2c3ca1b83161823c5a4519dda213fa9f1b69333754a077d10e9cf08b Nov 02 14:05:51.340122 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.340087 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a-tls-cert\") pod \"authorino-c75459655-fdx74\" (UID: \"60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a\") " pod="kuadrant-system/authorino-c75459655-fdx74" Nov 02 14:05:51.340325 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.340195 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lsdb\" (UniqueName: \"kubernetes.io/projected/60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a-kube-api-access-9lsdb\") pod \"authorino-c75459655-fdx74\" (UID: \"60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a\") " pod="kuadrant-system/authorino-c75459655-fdx74" Nov 02 14:05:51.342796 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.342764 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a-tls-cert\") pod \"authorino-c75459655-fdx74\" (UID: \"60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a\") " pod="kuadrant-system/authorino-c75459655-fdx74" Nov 02 14:05:51.369072 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.369044 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lsdb\" (UniqueName: \"kubernetes.io/projected/60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a-kube-api-access-9lsdb\") pod \"authorino-c75459655-fdx74\" (UID: \"60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a\") " pod="kuadrant-system/authorino-c75459655-fdx74" Nov 02 14:05:51.544849 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.544814 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c75459655-fdx74" Nov 02 14:05:51.603016 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.602990 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr"] Nov 02 14:05:51.605862 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:05:51.605837 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f38a69_4298_4c7c_a287_190d35223473.slice/crio-24e084e5e65ede40f6d8d799f32255c7079c821b8c6e0aa7de9a8ef861bc1697 WatchSource:0}: Error finding container 24e084e5e65ede40f6d8d799f32255c7079c821b8c6e0aa7de9a8ef861bc1697: Status 404 returned error can't find the container with id 24e084e5e65ede40f6d8d799f32255c7079c821b8c6e0aa7de9a8ef861bc1697 Nov 02 14:05:51.841326 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.841159 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c75459655-fdx74"] Nov 02 14:05:51.844106 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:05:51.844080 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60fcc00b_b8c8_4eff_8cf3_ffb70b0fab3a.slice/crio-1770341ada5942f924a99450c7ed353af1bf0a6d67474c182447bfd4bbca5626 WatchSource:0}: Error finding container 1770341ada5942f924a99450c7ed353af1bf0a6d67474c182447bfd4bbca5626: Status 404 returned error can't find the container with id 1770341ada5942f924a99450c7ed353af1bf0a6d67474c182447bfd4bbca5626 Nov 02 14:05:51.992160 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.992110 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" event={"ID":"a1f38a69-4298-4c7c-a287-190d35223473","Type":"ContainerStarted","Data":"24e084e5e65ede40f6d8d799f32255c7079c821b8c6e0aa7de9a8ef861bc1697"} Nov 02 14:05:51.993489 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.993451 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hv5z8" event={"ID":"43378270-dfc5-482c-aaec-97eb49a47481","Type":"ContainerStarted","Data":"8065343c2c3ca1b83161823c5a4519dda213fa9f1b69333754a077d10e9cf08b"} Nov 02 14:05:51.994850 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.994820 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c75459655-fdx74" event={"ID":"60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a","Type":"ContainerStarted","Data":"1770341ada5942f924a99450c7ed353af1bf0a6d67474c182447bfd4bbca5626"} Nov 02 14:05:51.994977 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:51.994851 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7696f" Nov 02 14:05:52.001916 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:52.001893 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7696f" Nov 02 14:05:52.045546 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:52.045513 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4-tls-cert\" (UniqueName: \"kubernetes.io/secret/eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4-tls-cert\") pod \"eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4\" (UID: \"eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4\") " Nov 02 14:05:52.045770 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:52.045558 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4-kube-api-access-k8hks\" (UniqueName: \"kubernetes.io/projected/eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4-kube-api-access-k8hks\") pod \"eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4\" (UID: \"eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4\") " Nov 02 14:05:52.048734 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:52.048348 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4-tls-cert" pod "eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4" (UID: "eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:05:52.048804 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:52.048733 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4-kube-api-access-k8hks" pod "eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4" (UID: "eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4"). InnerVolumeSpecName "kube-api-access-k8hks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:05:52.147348 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:52.147213 4866 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4-tls-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:05:52.147348 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:52.147246 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8hks\" (UniqueName: \"kubernetes.io/projected/eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4-kube-api-access-k8hks\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:05:53.003106 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:53.003072 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7696f" Nov 02 14:05:53.003565 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:53.003072 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" event={"ID":"a1f38a69-4298-4c7c-a287-190d35223473","Type":"ContainerStarted","Data":"160b6b2f96efd31b52d4726ed54973061dbedd6ba818aae0d315b095136c0a42"} Nov 02 14:05:53.003565 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:53.003410 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" Nov 02 14:05:53.049715 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:53.049654 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" podStartSLOduration=2.645727896 podStartE2EDuration="3.049633887s" totalImagesPullingTime="403.905991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:53.047592995 +0000 UTC m=+518.231440356" watchObservedRunningTime="2026-11-02 14:05:53.049633887 +0000 UTC m=+518.233481226" Nov 02 14:05:53.129770 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:53.129736 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7696f"] Nov 02 14:05:53.138238 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:53.137605 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7696f"] Nov 02 14:05:54.950745 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:54.950705 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4" path="/var/lib/kubelet/pods/eeff9db7-4703-4a5a-b9b4-1a5f6d8e3fc4/volumes" Nov 02 14:05:55.013973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:55.013918 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c75459655-fdx74" event={"ID":"60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a","Type":"ContainerStarted","Data":"6cae6ac413e6a50efbfd35cf64d17bbc627e121dee95a8fafe7218361ed2ee98"} Nov 02 14:05:55.015738 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:55.015712 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hv5z8" event={"ID":"43378270-dfc5-482c-aaec-97eb49a47481","Type":"ContainerStarted","Data":"6392f1d662b11d15b47f001fdeb967cc075ec0bac1de940f8b14484def9d1556"} Nov 02 14:05:55.015889 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:55.015851 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-hv5z8" podUID="43378270-dfc5-482c-aaec-97eb49a47481" containerName="authorino" containerID="cri-o://6392f1d662b11d15b47f001fdeb967cc075ec0bac1de940f8b14484def9d1556" gracePeriod=30 Nov 02 14:05:55.058347 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:55.058283 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-c75459655-fdx74" podStartSLOduration=1.429553578 podStartE2EDuration="4.058260835s" totalImagesPullingTime="2.628707257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:55.057212625 +0000 UTC m=+520.241059964" watchObservedRunningTime="2026-11-02 14:05:55.058260835 +0000 UTC m=+520.242108171" Nov 02 14:05:55.097663 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:55.097601 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-hv5z8" podStartSLOduration=2.235817448 podStartE2EDuration="5.096900343s" totalImagesPullingTime="2.861082895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:55.096388911 +0000 UTC m=+520.280236243" watchObservedRunningTime="2026-11-02 14:05:55.096900343 +0000 UTC m=+520.280747677" Nov 02 14:05:55.277874 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:55.277847 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hv5z8" Nov 02 14:05:55.389581 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:55.389542 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43378270-dfc5-482c-aaec-97eb49a47481-kube-api-access-dhbvw\" (UniqueName: \"kubernetes.io/projected/43378270-dfc5-482c-aaec-97eb49a47481-kube-api-access-dhbvw\") pod \"43378270-dfc5-482c-aaec-97eb49a47481\" (UID: \"43378270-dfc5-482c-aaec-97eb49a47481\") " Nov 02 14:05:55.391769 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:55.391737 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43378270-dfc5-482c-aaec-97eb49a47481-kube-api-access-dhbvw" pod "43378270-dfc5-482c-aaec-97eb49a47481" (UID: "43378270-dfc5-482c-aaec-97eb49a47481"). InnerVolumeSpecName "kube-api-access-dhbvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:05:55.490698 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:55.490655 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhbvw\" (UniqueName: \"kubernetes.io/projected/43378270-dfc5-482c-aaec-97eb49a47481-kube-api-access-dhbvw\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:05:56.020300 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:56.020263 4866 generic.go:356] "Generic (PLEG): container finished" podID="43378270-dfc5-482c-aaec-97eb49a47481" containerID="6392f1d662b11d15b47f001fdeb967cc075ec0bac1de940f8b14484def9d1556" exitCode=0 Nov 02 14:05:56.020710 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:56.020309 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hv5z8" Nov 02 14:05:56.020710 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:56.020346 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hv5z8" event={"ID":"43378270-dfc5-482c-aaec-97eb49a47481","Type":"ContainerDied","Data":"6392f1d662b11d15b47f001fdeb967cc075ec0bac1de940f8b14484def9d1556"} Nov 02 14:05:56.020710 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:56.020401 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hv5z8" event={"ID":"43378270-dfc5-482c-aaec-97eb49a47481","Type":"ContainerDied","Data":"8065343c2c3ca1b83161823c5a4519dda213fa9f1b69333754a077d10e9cf08b"} Nov 02 14:05:56.020710 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:56.020431 4866 scope.go:122] "RemoveContainer" containerID="6392f1d662b11d15b47f001fdeb967cc075ec0bac1de940f8b14484def9d1556" Nov 02 14:05:56.028938 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:56.028921 4866 scope.go:122] "RemoveContainer" containerID="6392f1d662b11d15b47f001fdeb967cc075ec0bac1de940f8b14484def9d1556" Nov 02 14:05:56.029238 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:05:56.029218 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6392f1d662b11d15b47f001fdeb967cc075ec0bac1de940f8b14484def9d1556\": container with ID starting with 6392f1d662b11d15b47f001fdeb967cc075ec0bac1de940f8b14484def9d1556 not found: ID does not exist" containerID="6392f1d662b11d15b47f001fdeb967cc075ec0bac1de940f8b14484def9d1556" Nov 02 14:05:56.029280 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:56.029247 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6392f1d662b11d15b47f001fdeb967cc075ec0bac1de940f8b14484def9d1556"} err="failed to get container status \"6392f1d662b11d15b47f001fdeb967cc075ec0bac1de940f8b14484def9d1556\": rpc error: code = NotFound desc = could not find container \"6392f1d662b11d15b47f001fdeb967cc075ec0bac1de940f8b14484def9d1556\": container with ID starting with 6392f1d662b11d15b47f001fdeb967cc075ec0bac1de940f8b14484def9d1556 not found: ID does not exist" Nov 02 14:05:56.073666 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:56.073633 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hv5z8"] Nov 02 14:05:56.080529 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:56.080501 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hv5z8"] Nov 02 14:05:56.944007 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:56.943973 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43378270-dfc5-482c-aaec-97eb49a47481" path="/var/lib/kubelet/pods/43378270-dfc5-482c-aaec-97eb49a47481/volumes" Nov 02 14:05:59.013411 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:59.013377 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" Nov 02 14:05:59.097315 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:59.097053 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-54889d7857-j65vr"] Nov 02 14:05:59.097315 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:59.097298 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" podUID="a600929e-ff62-427a-89dc-9f1eb7f71256" containerName="maas-api" containerID="cri-o://49dc5d7ea88637f80fcce1fc7af788f58640104252d3ace4231840720259b131" gracePeriod=30 Nov 02 14:05:59.337425 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:59.337400 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" Nov 02 14:05:59.426141 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:59.426109 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a600929e-ff62-427a-89dc-9f1eb7f71256-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a600929e-ff62-427a-89dc-9f1eb7f71256-metrics-certs\") pod \"a600929e-ff62-427a-89dc-9f1eb7f71256\" (UID: \"a600929e-ff62-427a-89dc-9f1eb7f71256\") " Nov 02 14:05:59.426310 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:59.426165 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a600929e-ff62-427a-89dc-9f1eb7f71256-kube-api-access-6h845\" (UniqueName: \"kubernetes.io/projected/a600929e-ff62-427a-89dc-9f1eb7f71256-kube-api-access-6h845\") pod \"a600929e-ff62-427a-89dc-9f1eb7f71256\" (UID: \"a600929e-ff62-427a-89dc-9f1eb7f71256\") " Nov 02 14:05:59.426310 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:59.426201 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a600929e-ff62-427a-89dc-9f1eb7f71256-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a600929e-ff62-427a-89dc-9f1eb7f71256-openshift-service-ca\") pod \"a600929e-ff62-427a-89dc-9f1eb7f71256\" (UID: \"a600929e-ff62-427a-89dc-9f1eb7f71256\") " Nov 02 14:05:59.426384 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:59.426315 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a600929e-ff62-427a-89dc-9f1eb7f71256-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a600929e-ff62-427a-89dc-9f1eb7f71256-maas-api-tls\") pod \"a600929e-ff62-427a-89dc-9f1eb7f71256\" (UID: \"a600929e-ff62-427a-89dc-9f1eb7f71256\") " Nov 02 14:05:59.426653 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:59.426619 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a600929e-ff62-427a-89dc-9f1eb7f71256-openshift-service-ca" pod "a600929e-ff62-427a-89dc-9f1eb7f71256" (UID: "a600929e-ff62-427a-89dc-9f1eb7f71256"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:05:59.428459 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:59.428429 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a600929e-ff62-427a-89dc-9f1eb7f71256-kube-api-access-6h845" pod "a600929e-ff62-427a-89dc-9f1eb7f71256" (UID: "a600929e-ff62-427a-89dc-9f1eb7f71256"). InnerVolumeSpecName "kube-api-access-6h845". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:05:59.428546 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:59.428475 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a600929e-ff62-427a-89dc-9f1eb7f71256-metrics-certs" pod "a600929e-ff62-427a-89dc-9f1eb7f71256" (UID: "a600929e-ff62-427a-89dc-9f1eb7f71256"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:05:59.428546 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:59.428495 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a600929e-ff62-427a-89dc-9f1eb7f71256-maas-api-tls" pod "a600929e-ff62-427a-89dc-9f1eb7f71256" (UID: "a600929e-ff62-427a-89dc-9f1eb7f71256"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:05:59.527878 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:59.527840 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a600929e-ff62-427a-89dc-9f1eb7f71256-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:05:59.527878 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:59.527870 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a600929e-ff62-427a-89dc-9f1eb7f71256-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:05:59.527878 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:59.527882 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h845\" (UniqueName: \"kubernetes.io/projected/a600929e-ff62-427a-89dc-9f1eb7f71256-kube-api-access-6h845\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:05:59.528134 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:05:59.527892 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a600929e-ff62-427a-89dc-9f1eb7f71256-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:06:00.037287 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:00.037251 4866 generic.go:356] "Generic (PLEG): container finished" podID="a600929e-ff62-427a-89dc-9f1eb7f71256" containerID="49dc5d7ea88637f80fcce1fc7af788f58640104252d3ace4231840720259b131" exitCode=0 Nov 02 14:06:00.037721 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:00.037304 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" Nov 02 14:06:00.037721 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:00.037331 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" event={"ID":"a600929e-ff62-427a-89dc-9f1eb7f71256","Type":"ContainerDied","Data":"49dc5d7ea88637f80fcce1fc7af788f58640104252d3ace4231840720259b131"} Nov 02 14:06:00.037721 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:00.037385 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54889d7857-j65vr" event={"ID":"a600929e-ff62-427a-89dc-9f1eb7f71256","Type":"ContainerDied","Data":"5dbf063ab65916fc9ad7af3bfecdaeeb6017cf2ba1d537132c2f48ef7a38fe43"} Nov 02 14:06:00.037721 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:00.037401 4866 scope.go:122] "RemoveContainer" containerID="49dc5d7ea88637f80fcce1fc7af788f58640104252d3ace4231840720259b131" Nov 02 14:06:00.046141 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:00.046121 4866 scope.go:122] "RemoveContainer" containerID="49dc5d7ea88637f80fcce1fc7af788f58640104252d3ace4231840720259b131" Nov 02 14:06:00.046422 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:06:00.046405 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49dc5d7ea88637f80fcce1fc7af788f58640104252d3ace4231840720259b131\": container with ID starting with 49dc5d7ea88637f80fcce1fc7af788f58640104252d3ace4231840720259b131 not found: ID does not exist" containerID="49dc5d7ea88637f80fcce1fc7af788f58640104252d3ace4231840720259b131" Nov 02 14:06:00.046468 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:00.046431 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49dc5d7ea88637f80fcce1fc7af788f58640104252d3ace4231840720259b131"} err="failed to get container status \"49dc5d7ea88637f80fcce1fc7af788f58640104252d3ace4231840720259b131\": rpc error: code = NotFound desc = could not find container \"49dc5d7ea88637f80fcce1fc7af788f58640104252d3ace4231840720259b131\": container with ID starting with 49dc5d7ea88637f80fcce1fc7af788f58640104252d3ace4231840720259b131 not found: ID does not exist" Nov 02 14:06:00.092514 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:00.092481 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-54889d7857-j65vr"] Nov 02 14:06:00.104549 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:00.104521 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-54889d7857-j65vr"] Nov 02 14:06:00.943936 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:00.943905 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a600929e-ff62-427a-89dc-9f1eb7f71256" path="/var/lib/kubelet/pods/a600929e-ff62-427a-89dc-9f1eb7f71256/volumes" Nov 02 14:06:06.009039 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:06:06.009004 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:06:29.738478 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:29.738445 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-xjj8d"] Nov 02 14:06:29.739009 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:29.738823 4866 cpu_manager.go:414] "containerMap: removing container" podUID="a600929e-ff62-427a-89dc-9f1eb7f71256" containerName="maas-api" Nov 02 14:06:29.739009 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:29.738836 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a600929e-ff62-427a-89dc-9f1eb7f71256" containerName="maas-api" Nov 02 14:06:29.739009 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:29.738846 4866 cpu_manager.go:414] "containerMap: removing container" podUID="43378270-dfc5-482c-aaec-97eb49a47481" containerName="authorino" Nov 02 14:06:29.739009 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:29.738851 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43378270-dfc5-482c-aaec-97eb49a47481" containerName="authorino" Nov 02 14:06:29.739009 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:29.738927 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="a600929e-ff62-427a-89dc-9f1eb7f71256" containerName="maas-api" Nov 02 14:06:29.739009 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:29.738935 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="43378270-dfc5-482c-aaec-97eb49a47481" containerName="authorino" Nov 02 14:06:29.743764 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:29.743745 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-xjj8d" Nov 02 14:06:29.762708 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:29.762685 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 14:06:29.772446 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:29.772412 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 14:06:29.772623 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:29.772609 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-dbn4h\"" Nov 02 14:06:29.779857 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:29.779835 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-xjj8d"] Nov 02 14:06:29.783869 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:29.783849 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f78d05f4-d8fe-42ea-a289-046ea5f2cbde-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-xjj8d\" (UID: \"f78d05f4-d8fe-42ea-a289-046ea5f2cbde\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-xjj8d" Nov 02 14:06:29.783987 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:29.783905 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmkzs\" (UniqueName: \"kubernetes.io/projected/f78d05f4-d8fe-42ea-a289-046ea5f2cbde-kube-api-access-hmkzs\") pod \"ai-gateway-controller-5d8dddb676-xjj8d\" (UID: \"f78d05f4-d8fe-42ea-a289-046ea5f2cbde\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-xjj8d" Nov 02 14:06:29.884863 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:29.884831 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmkzs\" (UniqueName: \"kubernetes.io/projected/f78d05f4-d8fe-42ea-a289-046ea5f2cbde-kube-api-access-hmkzs\") pod \"ai-gateway-controller-5d8dddb676-xjj8d\" (UID: \"f78d05f4-d8fe-42ea-a289-046ea5f2cbde\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-xjj8d" Nov 02 14:06:29.885070 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:29.884880 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f78d05f4-d8fe-42ea-a289-046ea5f2cbde-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-xjj8d\" (UID: \"f78d05f4-d8fe-42ea-a289-046ea5f2cbde\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-xjj8d" Nov 02 14:06:29.887343 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:29.887317 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f78d05f4-d8fe-42ea-a289-046ea5f2cbde-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-xjj8d\" (UID: \"f78d05f4-d8fe-42ea-a289-046ea5f2cbde\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-xjj8d" Nov 02 14:06:29.921434 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:29.921407 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmkzs\" (UniqueName: \"kubernetes.io/projected/f78d05f4-d8fe-42ea-a289-046ea5f2cbde-kube-api-access-hmkzs\") pod \"ai-gateway-controller-5d8dddb676-xjj8d\" (UID: \"f78d05f4-d8fe-42ea-a289-046ea5f2cbde\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-xjj8d" Nov 02 14:06:30.056559 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:30.056473 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-xjj8d" Nov 02 14:06:30.356367 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:30.356287 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-xjj8d"] Nov 02 14:06:30.359753 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:06:30.359727 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78d05f4_d8fe_42ea_a289_046ea5f2cbde.slice/crio-4199bbc19f1a88507cd38d8c75c9d74d087261168d8eee3f4cfc5b9a505c4fa3 WatchSource:0}: Error finding container 4199bbc19f1a88507cd38d8c75c9d74d087261168d8eee3f4cfc5b9a505c4fa3: Status 404 returned error can't find the container with id 4199bbc19f1a88507cd38d8c75c9d74d087261168d8eee3f4cfc5b9a505c4fa3 Nov 02 14:06:31.162358 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:31.162322 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-xjj8d" event={"ID":"f78d05f4-d8fe-42ea-a289-046ea5f2cbde","Type":"ContainerStarted","Data":"4199bbc19f1a88507cd38d8c75c9d74d087261168d8eee3f4cfc5b9a505c4fa3"} Nov 02 14:06:32.167426 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:32.167346 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-xjj8d" event={"ID":"f78d05f4-d8fe-42ea-a289-046ea5f2cbde","Type":"ContainerStarted","Data":"df2b3c005e317d081e051d994cd13da64735a91db6387b761c1876026449a0cc"} Nov 02 14:06:32.167786 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:32.167473 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-xjj8d" Nov 02 14:06:32.208995 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:32.208931 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-xjj8d" podStartSLOduration=1.674146426 podStartE2EDuration="3.20891613s" totalImagesPullingTime="1.534769704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:32.208528773 +0000 UTC m=+557.392376109" watchObservedRunningTime="2026-11-02 14:06:32.20891613 +0000 UTC m=+557.392763466" Nov 02 14:06:34.370318 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.370280 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-2btkh"] Nov 02 14:06:34.373819 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.373800 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2btkh" Nov 02 14:06:34.418129 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.418103 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 14:06:34.418129 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.418116 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-v4g45\"" Nov 02 14:06:34.419549 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.419531 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-2btkh"] Nov 02 14:06:34.424648 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.424629 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edd944f-87df-46e3-aeeb-9873603e1178-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-2btkh\" (UID: \"3edd944f-87df-46e3-aeeb-9873603e1178\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2btkh" Nov 02 14:06:34.424713 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.424674 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7zkc\" (UniqueName: \"kubernetes.io/projected/3edd944f-87df-46e3-aeeb-9873603e1178-kube-api-access-b7zkc\") pod \"payload-pre-processing-7b98bb678c-2btkh\" (UID: \"3edd944f-87df-46e3-aeeb-9873603e1178\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2btkh" Nov 02 14:06:34.453083 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.453053 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rx5hk"] Nov 02 14:06:34.456389 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.456368 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx5hk" Nov 02 14:06:34.479594 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.479563 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rx5hk"] Nov 02 14:06:34.525160 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.525130 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7zkc\" (UniqueName: \"kubernetes.io/projected/3edd944f-87df-46e3-aeeb-9873603e1178-kube-api-access-b7zkc\") pod \"payload-pre-processing-7b98bb678c-2btkh\" (UID: \"3edd944f-87df-46e3-aeeb-9873603e1178\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2btkh" Nov 02 14:06:34.525341 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.525218 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhsj6\" (UniqueName: \"kubernetes.io/projected/ce41c5ab-a5da-44b5-bf3e-f045c70f55ce-kube-api-access-lhsj6\") pod \"payload-processing-55fbbbf9dc-rx5hk\" (UID: \"ce41c5ab-a5da-44b5-bf3e-f045c70f55ce\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rx5hk" Nov 02 14:06:34.525341 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.525254 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edd944f-87df-46e3-aeeb-9873603e1178-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-2btkh\" (UID: \"3edd944f-87df-46e3-aeeb-9873603e1178\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2btkh" Nov 02 14:06:34.525341 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.525318 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce41c5ab-a5da-44b5-bf3e-f045c70f55ce-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rx5hk\" (UID: \"ce41c5ab-a5da-44b5-bf3e-f045c70f55ce\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rx5hk" Nov 02 14:06:34.525941 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.525924 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edd944f-87df-46e3-aeeb-9873603e1178-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-2btkh\" (UID: \"3edd944f-87df-46e3-aeeb-9873603e1178\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2btkh" Nov 02 14:06:34.549350 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.549315 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7zkc\" (UniqueName: \"kubernetes.io/projected/3edd944f-87df-46e3-aeeb-9873603e1178-kube-api-access-b7zkc\") pod \"payload-pre-processing-7b98bb678c-2btkh\" (UID: \"3edd944f-87df-46e3-aeeb-9873603e1178\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2btkh" Nov 02 14:06:34.626336 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.626246 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhsj6\" (UniqueName: \"kubernetes.io/projected/ce41c5ab-a5da-44b5-bf3e-f045c70f55ce-kube-api-access-lhsj6\") pod \"payload-processing-55fbbbf9dc-rx5hk\" (UID: \"ce41c5ab-a5da-44b5-bf3e-f045c70f55ce\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rx5hk" Nov 02 14:06:34.626336 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.626297 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce41c5ab-a5da-44b5-bf3e-f045c70f55ce-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rx5hk\" (UID: \"ce41c5ab-a5da-44b5-bf3e-f045c70f55ce\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rx5hk" Nov 02 14:06:34.627041 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.627020 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce41c5ab-a5da-44b5-bf3e-f045c70f55ce-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rx5hk\" (UID: \"ce41c5ab-a5da-44b5-bf3e-f045c70f55ce\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rx5hk" Nov 02 14:06:34.653436 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.653403 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhsj6\" (UniqueName: \"kubernetes.io/projected/ce41c5ab-a5da-44b5-bf3e-f045c70f55ce-kube-api-access-lhsj6\") pod \"payload-processing-55fbbbf9dc-rx5hk\" (UID: \"ce41c5ab-a5da-44b5-bf3e-f045c70f55ce\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rx5hk" Nov 02 14:06:34.686763 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.686734 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2btkh" Nov 02 14:06:34.769101 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.769060 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx5hk" Nov 02 14:06:34.982485 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:34.982343 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-2btkh"] Nov 02 14:06:34.984660 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:06:34.984625 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edd944f_87df_46e3_aeeb_9873603e1178.slice/crio-e1a098af69fcfaa939a4bbeaf57616f25c526aa97a30304c8794ce92b517c8cd WatchSource:0}: Error finding container e1a098af69fcfaa939a4bbeaf57616f25c526aa97a30304c8794ce92b517c8cd: Status 404 returned error can't find the container with id e1a098af69fcfaa939a4bbeaf57616f25c526aa97a30304c8794ce92b517c8cd Nov 02 14:06:35.064070 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:35.064040 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rx5hk"] Nov 02 14:06:35.067145 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:06:35.067112 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce41c5ab_a5da_44b5_bf3e_f045c70f55ce.slice/crio-0da2f80dc0c12786e964995393e1eeb64d232800091ed76a974b1badfe8958a2 WatchSource:0}: Error finding container 0da2f80dc0c12786e964995393e1eeb64d232800091ed76a974b1badfe8958a2: Status 404 returned error can't find the container with id 0da2f80dc0c12786e964995393e1eeb64d232800091ed76a974b1badfe8958a2 Nov 02 14:06:35.182681 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:35.182649 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2btkh" event={"ID":"3edd944f-87df-46e3-aeeb-9873603e1178","Type":"ContainerStarted","Data":"e1a098af69fcfaa939a4bbeaf57616f25c526aa97a30304c8794ce92b517c8cd"} Nov 02 14:06:35.183798 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:35.183772 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx5hk" event={"ID":"ce41c5ab-a5da-44b5-bf3e-f045c70f55ce","Type":"ContainerStarted","Data":"0da2f80dc0c12786e964995393e1eeb64d232800091ed76a974b1badfe8958a2"} Nov 02 14:06:36.017484 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:06:36.017376 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:06:38.204277 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:38.204233 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2btkh" event={"ID":"3edd944f-87df-46e3-aeeb-9873603e1178","Type":"ContainerStarted","Data":"9991431be31da33010d1512fe0fd70617a7c0b2d3ac0e865f6c80d93d20eb377"} Nov 02 14:06:38.204767 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:38.204362 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2btkh" Nov 02 14:06:38.205828 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:38.205804 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx5hk" event={"ID":"ce41c5ab-a5da-44b5-bf3e-f045c70f55ce","Type":"ContainerStarted","Data":"da39a05ebdf68e22e0121a37b53694cc73f15b44fc8eabe10aede7b080381abe"} Nov 02 14:06:38.205965 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:38.205861 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx5hk" Nov 02 14:06:38.246002 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:38.245934 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2btkh" podStartSLOduration=1.882982473 podStartE2EDuration="4.245923195s" totalImagesPullingTime="2.362940722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:38.245742884 +0000 UTC m=+563.429590222" watchObservedRunningTime="2026-11-02 14:06:38.245923195 +0000 UTC m=+563.429770559" Nov 02 14:06:38.288471 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:38.288406 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx5hk" podStartSLOduration=2.064789842 podStartE2EDuration="4.288387199s" totalImagesPullingTime="2.223597357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:38.287070015 +0000 UTC m=+563.470917353" watchObservedRunningTime="2026-11-02 14:06:38.288387199 +0000 UTC m=+563.472234538" Nov 02 14:06:43.176012 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:43.175976 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-xjj8d" Nov 02 14:06:43.467171 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:43.467137 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6cc7b895d5-5ldn8"] Nov 02 14:06:43.470903 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:43.470886 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cc7b895d5-5ldn8" Nov 02 14:06:43.493383 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:43.493354 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cc7b895d5-5ldn8"] Nov 02 14:06:43.504551 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:43.504517 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnljc\" (UniqueName: \"kubernetes.io/projected/4865da86-417f-4703-bcdc-c68b5de3ad59-kube-api-access-nnljc\") pod \"ai-gateway-controller-6cc7b895d5-5ldn8\" (UID: \"4865da86-417f-4703-bcdc-c68b5de3ad59\") " pod="opendatahub/ai-gateway-controller-6cc7b895d5-5ldn8" Nov 02 14:06:43.504678 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:43.504660 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4865da86-417f-4703-bcdc-c68b5de3ad59-metrics-certs\") pod \"ai-gateway-controller-6cc7b895d5-5ldn8\" (UID: \"4865da86-417f-4703-bcdc-c68b5de3ad59\") " pod="opendatahub/ai-gateway-controller-6cc7b895d5-5ldn8" Nov 02 14:06:43.606041 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:43.606005 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4865da86-417f-4703-bcdc-c68b5de3ad59-metrics-certs\") pod \"ai-gateway-controller-6cc7b895d5-5ldn8\" (UID: \"4865da86-417f-4703-bcdc-c68b5de3ad59\") " pod="opendatahub/ai-gateway-controller-6cc7b895d5-5ldn8" Nov 02 14:06:43.606209 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:43.606070 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnljc\" (UniqueName: \"kubernetes.io/projected/4865da86-417f-4703-bcdc-c68b5de3ad59-kube-api-access-nnljc\") pod \"ai-gateway-controller-6cc7b895d5-5ldn8\" (UID: \"4865da86-417f-4703-bcdc-c68b5de3ad59\") " pod="opendatahub/ai-gateway-controller-6cc7b895d5-5ldn8" Nov 02 14:06:43.608536 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:43.608506 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4865da86-417f-4703-bcdc-c68b5de3ad59-metrics-certs\") pod \"ai-gateway-controller-6cc7b895d5-5ldn8\" (UID: \"4865da86-417f-4703-bcdc-c68b5de3ad59\") " pod="opendatahub/ai-gateway-controller-6cc7b895d5-5ldn8" Nov 02 14:06:43.630010 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:43.629987 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnljc\" (UniqueName: \"kubernetes.io/projected/4865da86-417f-4703-bcdc-c68b5de3ad59-kube-api-access-nnljc\") pod \"ai-gateway-controller-6cc7b895d5-5ldn8\" (UID: \"4865da86-417f-4703-bcdc-c68b5de3ad59\") " pod="opendatahub/ai-gateway-controller-6cc7b895d5-5ldn8" Nov 02 14:06:43.783207 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:43.783117 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cc7b895d5-5ldn8" Nov 02 14:06:44.077566 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:44.077476 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cc7b895d5-5ldn8"] Nov 02 14:06:44.080780 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:06:44.080755 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4865da86_417f_4703_bcdc_c68b5de3ad59.slice/crio-80fcc804fe3e5afa77dfdeb8e081146057cdc58f1c23c62bc065711c80f5c9f6 WatchSource:0}: Error finding container 80fcc804fe3e5afa77dfdeb8e081146057cdc58f1c23c62bc065711c80f5c9f6: Status 404 returned error can't find the container with id 80fcc804fe3e5afa77dfdeb8e081146057cdc58f1c23c62bc065711c80f5c9f6 Nov 02 14:06:44.230351 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:44.230316 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cc7b895d5-5ldn8" event={"ID":"4865da86-417f-4703-bcdc-c68b5de3ad59","Type":"ContainerStarted","Data":"80fcc804fe3e5afa77dfdeb8e081146057cdc58f1c23c62bc065711c80f5c9f6"} Nov 02 14:06:45.235831 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:45.235790 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cc7b895d5-5ldn8" event={"ID":"4865da86-417f-4703-bcdc-c68b5de3ad59","Type":"ContainerStarted","Data":"092debb6f12f9a62f65e4098d9b64aa203a4fd6d75d1a41d85011345fd8b0ee9"} Nov 02 14:06:45.236264 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:45.235869 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6cc7b895d5-5ldn8" Nov 02 14:06:45.278872 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:45.278822 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6cc7b895d5-5ldn8" podStartSLOduration=1.843434393 podStartE2EDuration="2.278810503s" totalImagesPullingTime="435.37611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:45.278533457 +0000 UTC m=+570.462380797" watchObservedRunningTime="2026-11-02 14:06:45.278810503 +0000 UTC m=+570.462657840" Nov 02 14:06:49.213294 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:49.213265 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2btkh" Nov 02 14:06:49.213758 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:49.213347 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx5hk" Nov 02 14:06:56.246203 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:56.246175 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6cc7b895d5-5ldn8" Nov 02 14:06:56.317082 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:56.317045 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-xjj8d"] Nov 02 14:06:56.317381 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:56.317342 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-xjj8d" podUID="f78d05f4-d8fe-42ea-a289-046ea5f2cbde" containerName="manager" containerID="cri-o://df2b3c005e317d081e051d994cd13da64735a91db6387b761c1876026449a0cc" gracePeriod=30 Nov 02 14:06:56.558235 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:56.558211 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-xjj8d" Nov 02 14:06:56.620196 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:56.620165 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f78d05f4-d8fe-42ea-a289-046ea5f2cbde-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f78d05f4-d8fe-42ea-a289-046ea5f2cbde-metrics-certs\") pod \"f78d05f4-d8fe-42ea-a289-046ea5f2cbde\" (UID: \"f78d05f4-d8fe-42ea-a289-046ea5f2cbde\") " Nov 02 14:06:56.620358 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:56.620243 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f78d05f4-d8fe-42ea-a289-046ea5f2cbde-kube-api-access-hmkzs\" (UniqueName: \"kubernetes.io/projected/f78d05f4-d8fe-42ea-a289-046ea5f2cbde-kube-api-access-hmkzs\") pod \"f78d05f4-d8fe-42ea-a289-046ea5f2cbde\" (UID: \"f78d05f4-d8fe-42ea-a289-046ea5f2cbde\") " Nov 02 14:06:56.622365 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:56.622336 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f78d05f4-d8fe-42ea-a289-046ea5f2cbde-metrics-certs" pod "f78d05f4-d8fe-42ea-a289-046ea5f2cbde" (UID: "f78d05f4-d8fe-42ea-a289-046ea5f2cbde"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:06:56.622476 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:56.622419 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78d05f4-d8fe-42ea-a289-046ea5f2cbde-kube-api-access-hmkzs" pod "f78d05f4-d8fe-42ea-a289-046ea5f2cbde" (UID: "f78d05f4-d8fe-42ea-a289-046ea5f2cbde"). InnerVolumeSpecName "kube-api-access-hmkzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:06:56.721716 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:56.721682 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmkzs\" (UniqueName: \"kubernetes.io/projected/f78d05f4-d8fe-42ea-a289-046ea5f2cbde-kube-api-access-hmkzs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:06:56.721716 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:56.721712 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f78d05f4-d8fe-42ea-a289-046ea5f2cbde-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:06:57.288825 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:57.288738 4866 generic.go:356] "Generic (PLEG): container finished" podID="f78d05f4-d8fe-42ea-a289-046ea5f2cbde" containerID="df2b3c005e317d081e051d994cd13da64735a91db6387b761c1876026449a0cc" exitCode=0 Nov 02 14:06:57.288825 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:57.288783 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-xjj8d" Nov 02 14:06:57.289260 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:57.288783 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-xjj8d" event={"ID":"f78d05f4-d8fe-42ea-a289-046ea5f2cbde","Type":"ContainerDied","Data":"df2b3c005e317d081e051d994cd13da64735a91db6387b761c1876026449a0cc"} Nov 02 14:06:57.289260 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:57.288917 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-xjj8d" event={"ID":"f78d05f4-d8fe-42ea-a289-046ea5f2cbde","Type":"ContainerDied","Data":"4199bbc19f1a88507cd38d8c75c9d74d087261168d8eee3f4cfc5b9a505c4fa3"} Nov 02 14:06:57.289260 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:57.288970 4866 scope.go:122] "RemoveContainer" containerID="df2b3c005e317d081e051d994cd13da64735a91db6387b761c1876026449a0cc" Nov 02 14:06:57.297167 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:57.297145 4866 scope.go:122] "RemoveContainer" containerID="df2b3c005e317d081e051d994cd13da64735a91db6387b761c1876026449a0cc" Nov 02 14:06:57.297452 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:06:57.297434 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df2b3c005e317d081e051d994cd13da64735a91db6387b761c1876026449a0cc\": container with ID starting with df2b3c005e317d081e051d994cd13da64735a91db6387b761c1876026449a0cc not found: ID does not exist" containerID="df2b3c005e317d081e051d994cd13da64735a91db6387b761c1876026449a0cc" Nov 02 14:06:57.297496 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:57.297460 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df2b3c005e317d081e051d994cd13da64735a91db6387b761c1876026449a0cc"} err="failed to get container status \"df2b3c005e317d081e051d994cd13da64735a91db6387b761c1876026449a0cc\": rpc error: code = NotFound desc = could not find container \"df2b3c005e317d081e051d994cd13da64735a91db6387b761c1876026449a0cc\": container with ID starting with df2b3c005e317d081e051d994cd13da64735a91db6387b761c1876026449a0cc not found: ID does not exist" Nov 02 14:06:57.339172 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:57.339145 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-xjj8d"] Nov 02 14:06:57.350988 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:57.350940 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-xjj8d"] Nov 02 14:06:58.944344 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:06:58.944310 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f78d05f4-d8fe-42ea-a289-046ea5f2cbde" path="/var/lib/kubelet/pods/f78d05f4-d8fe-42ea-a289-046ea5f2cbde/volumes" Nov 02 14:07:06.007902 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:07:06.007863 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:07:14.974911 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:14.974880 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:07:14.976227 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:14.976206 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:07:20.873169 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:20.873135 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg"] Nov 02 14:07:20.874041 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:20.874012 4866 cpu_manager.go:414] "containerMap: removing container" podUID="f78d05f4-d8fe-42ea-a289-046ea5f2cbde" containerName="manager" Nov 02 14:07:20.874041 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:20.874034 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78d05f4-d8fe-42ea-a289-046ea5f2cbde" containerName="manager" Nov 02 14:07:20.874182 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:20.874103 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="f78d05f4-d8fe-42ea-a289-046ea5f2cbde" containerName="manager" Nov 02 14:07:20.881237 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:20.881213 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:20.899147 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:20.899126 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bm7wg\"" Nov 02 14:07:20.900332 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:20.900309 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg"] Nov 02 14:07:20.911216 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:20.911195 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 14:07:20.911216 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:20.911195 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 14:07:20.911348 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:20.911249 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 02 14:07:20.924668 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:20.924644 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8983367-2f54-4d0c-879a-bc768a47a1ee-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg\" (UID: \"c8983367-2f54-4d0c-879a-bc768a47a1ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:20.924771 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:20.924674 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8h5k\" (UniqueName: \"kubernetes.io/projected/c8983367-2f54-4d0c-879a-bc768a47a1ee-kube-api-access-k8h5k\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg\" (UID: \"c8983367-2f54-4d0c-879a-bc768a47a1ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:20.924771 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:20.924704 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8983367-2f54-4d0c-879a-bc768a47a1ee-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg\" (UID: \"c8983367-2f54-4d0c-879a-bc768a47a1ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:20.924771 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:20.924756 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8983367-2f54-4d0c-879a-bc768a47a1ee-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg\" (UID: \"c8983367-2f54-4d0c-879a-bc768a47a1ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:20.924909 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:20.924891 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8983367-2f54-4d0c-879a-bc768a47a1ee-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg\" (UID: \"c8983367-2f54-4d0c-879a-bc768a47a1ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:20.924968 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:20.924930 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8983367-2f54-4d0c-879a-bc768a47a1ee-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg\" (UID: \"c8983367-2f54-4d0c-879a-bc768a47a1ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:21.025921 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:21.025862 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8983367-2f54-4d0c-879a-bc768a47a1ee-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg\" (UID: \"c8983367-2f54-4d0c-879a-bc768a47a1ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:21.026137 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:21.025934 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8983367-2f54-4d0c-879a-bc768a47a1ee-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg\" (UID: \"c8983367-2f54-4d0c-879a-bc768a47a1ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:21.026137 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:21.026005 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8983367-2f54-4d0c-879a-bc768a47a1ee-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg\" (UID: \"c8983367-2f54-4d0c-879a-bc768a47a1ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:21.026137 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:21.026029 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8h5k\" (UniqueName: \"kubernetes.io/projected/c8983367-2f54-4d0c-879a-bc768a47a1ee-kube-api-access-k8h5k\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg\" (UID: \"c8983367-2f54-4d0c-879a-bc768a47a1ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:21.026137 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:21.026073 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8983367-2f54-4d0c-879a-bc768a47a1ee-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg\" (UID: \"c8983367-2f54-4d0c-879a-bc768a47a1ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:21.026137 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:21.026111 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8983367-2f54-4d0c-879a-bc768a47a1ee-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg\" (UID: \"c8983367-2f54-4d0c-879a-bc768a47a1ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:21.026385 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:21.026349 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8983367-2f54-4d0c-879a-bc768a47a1ee-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg\" (UID: \"c8983367-2f54-4d0c-879a-bc768a47a1ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:21.026476 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:21.026452 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8983367-2f54-4d0c-879a-bc768a47a1ee-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg\" (UID: \"c8983367-2f54-4d0c-879a-bc768a47a1ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:21.026542 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:21.026511 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8983367-2f54-4d0c-879a-bc768a47a1ee-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg\" (UID: \"c8983367-2f54-4d0c-879a-bc768a47a1ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:21.028337 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:21.028317 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8983367-2f54-4d0c-879a-bc768a47a1ee-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg\" (UID: \"c8983367-2f54-4d0c-879a-bc768a47a1ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:21.028528 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:21.028510 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8983367-2f54-4d0c-879a-bc768a47a1ee-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg\" (UID: \"c8983367-2f54-4d0c-879a-bc768a47a1ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:21.050862 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:21.050827 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8h5k\" (UniqueName: \"kubernetes.io/projected/c8983367-2f54-4d0c-879a-bc768a47a1ee-kube-api-access-k8h5k\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg\" (UID: \"c8983367-2f54-4d0c-879a-bc768a47a1ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:21.193876 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:21.193841 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:21.490660 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:21.490604 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg"] Nov 02 14:07:21.493048 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:07:21.493020 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8983367_2f54_4d0c_879a_bc768a47a1ee.slice/crio-c77a72645634d5cf3fb3a0134cb41b68f4b9e229260243d4b06981235b4ed901 WatchSource:0}: Error finding container c77a72645634d5cf3fb3a0134cb41b68f4b9e229260243d4b06981235b4ed901: Status 404 returned error can't find the container with id c77a72645634d5cf3fb3a0134cb41b68f4b9e229260243d4b06981235b4ed901 Nov 02 14:07:22.392580 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:22.392537 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" event={"ID":"c8983367-2f54-4d0c-879a-bc768a47a1ee","Type":"ContainerStarted","Data":"c77a72645634d5cf3fb3a0134cb41b68f4b9e229260243d4b06981235b4ed901"} Nov 02 14:07:23.399885 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:23.399852 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" event={"ID":"c8983367-2f54-4d0c-879a-bc768a47a1ee","Type":"ContainerStarted","Data":"df2961ce75c88ffd1797a66b119cc997e8e4dca5e754367b2f5d5b144c55d58a"} Nov 02 14:07:23.446566 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:23.446503 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" podStartSLOduration=2.184357415 podStartE2EDuration="3.446488253s" totalImagesPullingTime="1.262130838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:23.444690778 +0000 UTC m=+608.628538115" watchObservedRunningTime="2026-11-02 14:07:23.446488253 +0000 UTC m=+608.630335591" Nov 02 14:07:23.875354 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:23.875313 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx"] Nov 02 14:07:23.878929 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:23.878906 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:23.898341 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:23.898318 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 02 14:07:23.908416 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:23.908390 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx"] Nov 02 14:07:23.946764 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:23.946727 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kwrdx\" (UID: \"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:23.946764 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:23.946763 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xg22\" (UniqueName: \"kubernetes.io/projected/d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102-kube-api-access-2xg22\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kwrdx\" (UID: \"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:23.947007 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:23.946786 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kwrdx\" (UID: \"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:23.947007 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:23.946900 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kwrdx\" (UID: \"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:23.947007 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:23.946938 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kwrdx\" (UID: \"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:23.947164 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:23.947048 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kwrdx\" (UID: \"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:24.047927 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:24.047892 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kwrdx\" (UID: \"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:24.048107 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:24.047935 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xg22\" (UniqueName: \"kubernetes.io/projected/d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102-kube-api-access-2xg22\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kwrdx\" (UID: \"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:24.048107 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:24.047988 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kwrdx\" (UID: \"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:24.048107 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:24.048013 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kwrdx\" (UID: \"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:24.048107 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:24.048033 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kwrdx\" (UID: \"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:24.048107 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:24.048061 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kwrdx\" (UID: \"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:24.048422 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:24.048395 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kwrdx\" (UID: \"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:24.048504 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:24.048443 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kwrdx\" (UID: \"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:24.048504 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:24.048480 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kwrdx\" (UID: \"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:24.050389 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:24.050360 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kwrdx\" (UID: \"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:24.050616 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:24.050592 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kwrdx\" (UID: \"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:24.090772 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:24.090738 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xg22\" (UniqueName: \"kubernetes.io/projected/d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102-kube-api-access-2xg22\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kwrdx\" (UID: \"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:24.194207 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:24.194150 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:24.194380 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:24.194226 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:24.494724 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:24.494690 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx"] Nov 02 14:07:24.495303 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:07:24.495278 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f9dd22_8f5b_477f_9ca6_3ea2a6e9b102.slice/crio-7507a6262fe23ced6106a4e21e3683e61418b0abe5b805c78b296057024b2913 WatchSource:0}: Error finding container 7507a6262fe23ced6106a4e21e3683e61418b0abe5b805c78b296057024b2913: Status 404 returned error can't find the container with id 7507a6262fe23ced6106a4e21e3683e61418b0abe5b805c78b296057024b2913 Nov 02 14:07:25.412543 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:25.412505 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" event={"ID":"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102","Type":"ContainerStarted","Data":"3d6e10f6554af4712cd0140d59e97556b511fafb5edb8a0f09533e6769d453bf"} Nov 02 14:07:25.412693 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:25.412561 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" event={"ID":"d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102","Type":"ContainerStarted","Data":"7507a6262fe23ced6106a4e21e3683e61418b0abe5b805c78b296057024b2913"} Nov 02 14:07:25.456737 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:25.456677 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" podStartSLOduration=2.253516785 podStartE2EDuration="2.456653018s" totalImagesPullingTime="203.136233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:25.455827393 +0000 UTC m=+610.639674740" watchObservedRunningTime="2026-11-02 14:07:25.456653018 +0000 UTC m=+610.640500356" Nov 02 14:07:26.195372 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:26.195331 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:28.773441 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.773398 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn"] Nov 02 14:07:28.777012 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.776996 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:28.805084 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.805057 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 02 14:07:28.806636 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.806610 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn"] Nov 02 14:07:28.891679 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.891646 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ba45fd0-b74d-4b8c-be7a-4d521845c7d4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nnfxn\" (UID: \"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:28.891848 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.891687 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba45fd0-b74d-4b8c-be7a-4d521845c7d4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nnfxn\" (UID: \"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:28.891848 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.891798 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ba45fd0-b74d-4b8c-be7a-4d521845c7d4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nnfxn\" (UID: \"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:28.891848 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.891816 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfj9m\" (UniqueName: \"kubernetes.io/projected/4ba45fd0-b74d-4b8c-be7a-4d521845c7d4-kube-api-access-kfj9m\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nnfxn\" (UID: \"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:28.892040 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.891851 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ba45fd0-b74d-4b8c-be7a-4d521845c7d4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nnfxn\" (UID: \"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:28.892040 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.891870 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba45fd0-b74d-4b8c-be7a-4d521845c7d4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nnfxn\" (UID: \"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:28.992577 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.992542 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ba45fd0-b74d-4b8c-be7a-4d521845c7d4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nnfxn\" (UID: \"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:28.992577 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.992578 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba45fd0-b74d-4b8c-be7a-4d521845c7d4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nnfxn\" (UID: \"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:28.992775 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.992601 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ba45fd0-b74d-4b8c-be7a-4d521845c7d4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nnfxn\" (UID: \"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:28.992775 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.992715 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba45fd0-b74d-4b8c-be7a-4d521845c7d4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nnfxn\" (UID: \"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:28.992855 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.992799 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ba45fd0-b74d-4b8c-be7a-4d521845c7d4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nnfxn\" (UID: \"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:28.992855 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.992830 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfj9m\" (UniqueName: \"kubernetes.io/projected/4ba45fd0-b74d-4b8c-be7a-4d521845c7d4-kube-api-access-kfj9m\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nnfxn\" (UID: \"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:28.992995 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.992971 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ba45fd0-b74d-4b8c-be7a-4d521845c7d4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nnfxn\" (UID: \"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:28.993115 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.993088 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba45fd0-b74d-4b8c-be7a-4d521845c7d4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nnfxn\" (UID: \"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:28.993252 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.993210 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ba45fd0-b74d-4b8c-be7a-4d521845c7d4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nnfxn\" (UID: \"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:28.994941 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.994922 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ba45fd0-b74d-4b8c-be7a-4d521845c7d4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nnfxn\" (UID: \"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:28.995367 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:28.995347 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba45fd0-b74d-4b8c-be7a-4d521845c7d4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nnfxn\" (UID: \"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:29.016526 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:29.016503 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfj9m\" (UniqueName: \"kubernetes.io/projected/4ba45fd0-b74d-4b8c-be7a-4d521845c7d4-kube-api-access-kfj9m\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nnfxn\" (UID: \"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:29.091764 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:29.091694 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:29.390097 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:29.390069 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn"] Nov 02 14:07:29.391884 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:07:29.391860 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba45fd0_b74d_4b8c_be7a_4d521845c7d4.slice/crio-62583437fcf97c4e29fcb56e27209eece623a50d852ece4a29a8e05d8cefc465 WatchSource:0}: Error finding container 62583437fcf97c4e29fcb56e27209eece623a50d852ece4a29a8e05d8cefc465: Status 404 returned error can't find the container with id 62583437fcf97c4e29fcb56e27209eece623a50d852ece4a29a8e05d8cefc465 Nov 02 14:07:29.439283 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:29.439251 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" event={"ID":"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4","Type":"ContainerStarted","Data":"62583437fcf97c4e29fcb56e27209eece623a50d852ece4a29a8e05d8cefc465"} Nov 02 14:07:30.444982 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:30.444936 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" event={"ID":"4ba45fd0-b74d-4b8c-be7a-4d521845c7d4","Type":"ContainerStarted","Data":"304805552ad105c736be2b97116c110db688423cd8e8af3d79714300685f914a"} Nov 02 14:07:30.491975 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:30.491155 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" podStartSLOduration=2.298931988 podStartE2EDuration="2.491141742s" totalImagesPullingTime="192.209754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:30.488484918 +0000 UTC m=+615.672332255" watchObservedRunningTime="2026-11-02 14:07:30.491141742 +0000 UTC m=+615.674989078" Nov 02 14:07:31.092022 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:31.091987 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:31.194887 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:31.194852 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:31.207777 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:31.207747 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:31.466530 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:31.460801 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg" Nov 02 14:07:32.576339 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.576301 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-92h45"] Nov 02 14:07:32.579988 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.579964 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:32.609374 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.609350 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 02 14:07:32.611030 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.611001 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-92h45"] Nov 02 14:07:32.633462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.633432 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ef4054-ab42-4598-a5e1-ac424fab8a8e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-92h45\" (UID: \"c9ef4054-ab42-4598-a5e1-ac424fab8a8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:32.633607 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.633468 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ef4054-ab42-4598-a5e1-ac424fab8a8e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-92h45\" (UID: \"c9ef4054-ab42-4598-a5e1-ac424fab8a8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:32.633607 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.633491 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ef4054-ab42-4598-a5e1-ac424fab8a8e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-92h45\" (UID: \"c9ef4054-ab42-4598-a5e1-ac424fab8a8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:32.633607 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.633577 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd9zh\" (UniqueName: \"kubernetes.io/projected/c9ef4054-ab42-4598-a5e1-ac424fab8a8e-kube-api-access-cd9zh\") pod \"echo-simulator-kserve-64b6f8f96c-92h45\" (UID: \"c9ef4054-ab42-4598-a5e1-ac424fab8a8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:32.633769 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.633653 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ef4054-ab42-4598-a5e1-ac424fab8a8e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-92h45\" (UID: \"c9ef4054-ab42-4598-a5e1-ac424fab8a8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:32.633769 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.633687 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ef4054-ab42-4598-a5e1-ac424fab8a8e-home\") pod \"echo-simulator-kserve-64b6f8f96c-92h45\" (UID: \"c9ef4054-ab42-4598-a5e1-ac424fab8a8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:32.734328 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.734288 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ef4054-ab42-4598-a5e1-ac424fab8a8e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-92h45\" (UID: \"c9ef4054-ab42-4598-a5e1-ac424fab8a8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:32.734328 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.734333 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ef4054-ab42-4598-a5e1-ac424fab8a8e-home\") pod \"echo-simulator-kserve-64b6f8f96c-92h45\" (UID: \"c9ef4054-ab42-4598-a5e1-ac424fab8a8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:32.734586 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.734405 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ef4054-ab42-4598-a5e1-ac424fab8a8e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-92h45\" (UID: \"c9ef4054-ab42-4598-a5e1-ac424fab8a8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:32.734586 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.734438 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ef4054-ab42-4598-a5e1-ac424fab8a8e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-92h45\" (UID: \"c9ef4054-ab42-4598-a5e1-ac424fab8a8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:32.734586 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.734469 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ef4054-ab42-4598-a5e1-ac424fab8a8e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-92h45\" (UID: \"c9ef4054-ab42-4598-a5e1-ac424fab8a8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:32.734586 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.734502 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd9zh\" (UniqueName: \"kubernetes.io/projected/c9ef4054-ab42-4598-a5e1-ac424fab8a8e-kube-api-access-cd9zh\") pod \"echo-simulator-kserve-64b6f8f96c-92h45\" (UID: \"c9ef4054-ab42-4598-a5e1-ac424fab8a8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:32.734911 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.734877 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ef4054-ab42-4598-a5e1-ac424fab8a8e-home\") pod \"echo-simulator-kserve-64b6f8f96c-92h45\" (UID: \"c9ef4054-ab42-4598-a5e1-ac424fab8a8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:32.734911 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.734903 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ef4054-ab42-4598-a5e1-ac424fab8a8e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-92h45\" (UID: \"c9ef4054-ab42-4598-a5e1-ac424fab8a8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:32.735105 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.735000 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ef4054-ab42-4598-a5e1-ac424fab8a8e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-92h45\" (UID: \"c9ef4054-ab42-4598-a5e1-ac424fab8a8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:32.736773 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.736737 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ef4054-ab42-4598-a5e1-ac424fab8a8e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-92h45\" (UID: \"c9ef4054-ab42-4598-a5e1-ac424fab8a8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:32.736974 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.736929 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ef4054-ab42-4598-a5e1-ac424fab8a8e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-92h45\" (UID: \"c9ef4054-ab42-4598-a5e1-ac424fab8a8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:32.759762 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.759724 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd9zh\" (UniqueName: \"kubernetes.io/projected/c9ef4054-ab42-4598-a5e1-ac424fab8a8e-kube-api-access-cd9zh\") pod \"echo-simulator-kserve-64b6f8f96c-92h45\" (UID: \"c9ef4054-ab42-4598-a5e1-ac424fab8a8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:32.893853 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:32.893737 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:33.213843 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:33.213812 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-92h45"] Nov 02 14:07:33.216712 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:07:33.216684 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ef4054_ab42_4598_a5e1_ac424fab8a8e.slice/crio-1babaef150a486fccf100d480972ea4423a0a21d55a6ca3560b3f01ca704dff4 WatchSource:0}: Error finding container 1babaef150a486fccf100d480972ea4423a0a21d55a6ca3560b3f01ca704dff4: Status 404 returned error can't find the container with id 1babaef150a486fccf100d480972ea4423a0a21d55a6ca3560b3f01ca704dff4 Nov 02 14:07:33.218036 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:33.218015 4866 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:07:33.464519 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:33.464432 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" event={"ID":"c9ef4054-ab42-4598-a5e1-ac424fab8a8e","Type":"ContainerStarted","Data":"1babaef150a486fccf100d480972ea4423a0a21d55a6ca3560b3f01ca704dff4"} Nov 02 14:07:34.196591 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:34.196556 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:34.210867 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:34.210797 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:34.486108 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:34.485921 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kwrdx" Nov 02 14:07:35.480989 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:35.480928 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" event={"ID":"c9ef4054-ab42-4598-a5e1-ac424fab8a8e","Type":"ContainerStarted","Data":"1d60dd5e61aace6dbd7d2d55047bc58383145a713c67db35d84467da843eefad"} Nov 02 14:07:35.535972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:35.530910 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" podStartSLOduration=1.715353481 podStartE2EDuration="3.530891893s" totalImagesPullingTime="1.815538412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:35.529176705 +0000 UTC m=+620.713024043" watchObservedRunningTime="2026-11-02 14:07:35.530891893 +0000 UTC m=+620.714739233" Nov 02 14:07:35.894987 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:35.894870 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:36.009082 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:07:36.009041 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:07:36.173760 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.173681 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv"] Nov 02 14:07:36.177382 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.177364 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.194151 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.194124 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 14:07:36.207615 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.207586 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv"] Nov 02 14:07:36.210146 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.210118 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4tfr8\"" Nov 02 14:07:36.274137 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.274098 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fbf701c-a1aa-4031-8112-4e69e00e103b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.274137 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.274138 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fbf701c-a1aa-4031-8112-4e69e00e103b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.274354 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.274216 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fbf701c-a1aa-4031-8112-4e69e00e103b-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.274354 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.274259 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fbf701c-a1aa-4031-8112-4e69e00e103b-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.274354 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.274308 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fbf701c-a1aa-4031-8112-4e69e00e103b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.274354 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.274326 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d9m7\" (UniqueName: \"kubernetes.io/projected/1fbf701c-a1aa-4031-8112-4e69e00e103b-kube-api-access-5d9m7\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.274481 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.274412 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fbf701c-a1aa-4031-8112-4e69e00e103b-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.274481 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.274431 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fbf701c-a1aa-4031-8112-4e69e00e103b-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.274481 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.274450 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fbf701c-a1aa-4031-8112-4e69e00e103b-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.374900 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.374859 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9m7\" (UniqueName: \"kubernetes.io/projected/1fbf701c-a1aa-4031-8112-4e69e00e103b-kube-api-access-5d9m7\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.375152 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.375018 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fbf701c-a1aa-4031-8112-4e69e00e103b-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.375152 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.375054 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fbf701c-a1aa-4031-8112-4e69e00e103b-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.375152 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.375079 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fbf701c-a1aa-4031-8112-4e69e00e103b-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.375152 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.375123 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fbf701c-a1aa-4031-8112-4e69e00e103b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.375152 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.375145 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fbf701c-a1aa-4031-8112-4e69e00e103b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.375434 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.375196 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fbf701c-a1aa-4031-8112-4e69e00e103b-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.375434 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.375387 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fbf701c-a1aa-4031-8112-4e69e00e103b-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.375537 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.375437 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fbf701c-a1aa-4031-8112-4e69e00e103b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.375589 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.375532 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fbf701c-a1aa-4031-8112-4e69e00e103b-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.375630 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.375582 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fbf701c-a1aa-4031-8112-4e69e00e103b-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.375673 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.375640 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fbf701c-a1aa-4031-8112-4e69e00e103b-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.375813 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.375787 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fbf701c-a1aa-4031-8112-4e69e00e103b-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.375890 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.375864 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fbf701c-a1aa-4031-8112-4e69e00e103b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.377599 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.377565 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fbf701c-a1aa-4031-8112-4e69e00e103b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.378007 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.377980 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fbf701c-a1aa-4031-8112-4e69e00e103b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.399615 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.399586 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9m7\" (UniqueName: \"kubernetes.io/projected/1fbf701c-a1aa-4031-8112-4e69e00e103b-kube-api-access-5d9m7\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.400937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.400915 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fbf701c-a1aa-4031-8112-4e69e00e103b-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-8q5hv\" (UID: \"1fbf701c-a1aa-4031-8112-4e69e00e103b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.491904 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.491868 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv"] Nov 02 14:07:36.492398 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.492024 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:36.496406 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.496254 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.512020 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.511987 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 02 14:07:36.522146 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.522116 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv"] Nov 02 14:07:36.576576 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.576541 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d89a28a-7f87-4924-93c3-48ae7d11e56e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv\" (UID: \"6d89a28a-7f87-4924-93c3-48ae7d11e56e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.576717 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.576628 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d89a28a-7f87-4924-93c3-48ae7d11e56e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv\" (UID: \"6d89a28a-7f87-4924-93c3-48ae7d11e56e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.576759 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.576711 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nv7l\" (UniqueName: \"kubernetes.io/projected/6d89a28a-7f87-4924-93c3-48ae7d11e56e-kube-api-access-4nv7l\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv\" (UID: \"6d89a28a-7f87-4924-93c3-48ae7d11e56e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.576806 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.576773 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d89a28a-7f87-4924-93c3-48ae7d11e56e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv\" (UID: \"6d89a28a-7f87-4924-93c3-48ae7d11e56e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.576840 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.576826 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d89a28a-7f87-4924-93c3-48ae7d11e56e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv\" (UID: \"6d89a28a-7f87-4924-93c3-48ae7d11e56e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.576875 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.576855 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d89a28a-7f87-4924-93c3-48ae7d11e56e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv\" (UID: \"6d89a28a-7f87-4924-93c3-48ae7d11e56e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.677499 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.677460 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d89a28a-7f87-4924-93c3-48ae7d11e56e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv\" (UID: \"6d89a28a-7f87-4924-93c3-48ae7d11e56e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.677700 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.677532 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nv7l\" (UniqueName: \"kubernetes.io/projected/6d89a28a-7f87-4924-93c3-48ae7d11e56e-kube-api-access-4nv7l\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv\" (UID: \"6d89a28a-7f87-4924-93c3-48ae7d11e56e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.677700 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.677564 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d89a28a-7f87-4924-93c3-48ae7d11e56e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv\" (UID: \"6d89a28a-7f87-4924-93c3-48ae7d11e56e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.677700 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.677584 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d89a28a-7f87-4924-93c3-48ae7d11e56e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv\" (UID: \"6d89a28a-7f87-4924-93c3-48ae7d11e56e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.677700 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.677601 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d89a28a-7f87-4924-93c3-48ae7d11e56e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv\" (UID: \"6d89a28a-7f87-4924-93c3-48ae7d11e56e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.677928 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.677749 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d89a28a-7f87-4924-93c3-48ae7d11e56e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv\" (UID: \"6d89a28a-7f87-4924-93c3-48ae7d11e56e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.678047 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.678025 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d89a28a-7f87-4924-93c3-48ae7d11e56e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv\" (UID: \"6d89a28a-7f87-4924-93c3-48ae7d11e56e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.678047 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.678041 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d89a28a-7f87-4924-93c3-48ae7d11e56e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv\" (UID: \"6d89a28a-7f87-4924-93c3-48ae7d11e56e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.678155 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.678086 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d89a28a-7f87-4924-93c3-48ae7d11e56e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv\" (UID: \"6d89a28a-7f87-4924-93c3-48ae7d11e56e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.679779 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.679761 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d89a28a-7f87-4924-93c3-48ae7d11e56e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv\" (UID: \"6d89a28a-7f87-4924-93c3-48ae7d11e56e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.679969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.679939 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d89a28a-7f87-4924-93c3-48ae7d11e56e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv\" (UID: \"6d89a28a-7f87-4924-93c3-48ae7d11e56e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.703352 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.703328 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nv7l\" (UniqueName: \"kubernetes.io/projected/6d89a28a-7f87-4924-93c3-48ae7d11e56e-kube-api-access-4nv7l\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv\" (UID: \"6d89a28a-7f87-4924-93c3-48ae7d11e56e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.827114 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.827020 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:36.831083 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:36.831062 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv"] Nov 02 14:07:36.833116 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:07:36.833084 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fbf701c_a1aa_4031_8112_4e69e00e103b.slice/crio-fd6504f1c437db341ef21d40fe6d6e38b2f1128ccc774306e3635a4c7cb6e119 WatchSource:0}: Error finding container fd6504f1c437db341ef21d40fe6d6e38b2f1128ccc774306e3635a4c7cb6e119: Status 404 returned error can't find the container with id fd6504f1c437db341ef21d40fe6d6e38b2f1128ccc774306e3635a4c7cb6e119 Nov 02 14:07:37.134044 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:37.133898 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv"] Nov 02 14:07:37.137581 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:07:37.137541 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d89a28a_7f87_4924_93c3_48ae7d11e56e.slice/crio-d282b398f8d2e4c187c06f645b851d1e034010cebcb3df31227a550049c9d314 WatchSource:0}: Error finding container d282b398f8d2e4c187c06f645b851d1e034010cebcb3df31227a550049c9d314: Status 404 returned error can't find the container with id d282b398f8d2e4c187c06f645b851d1e034010cebcb3df31227a550049c9d314 Nov 02 14:07:37.495589 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:37.495549 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" event={"ID":"6d89a28a-7f87-4924-93c3-48ae7d11e56e","Type":"ContainerStarted","Data":"20e25474732d3b2358189373e49ad53cf3b5a6383972fe45ee3842a7f9c6c745"} Nov 02 14:07:37.496044 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:37.495603 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" event={"ID":"6d89a28a-7f87-4924-93c3-48ae7d11e56e","Type":"ContainerStarted","Data":"d282b398f8d2e4c187c06f645b851d1e034010cebcb3df31227a550049c9d314"} Nov 02 14:07:37.497063 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:37.497037 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" event={"ID":"1fbf701c-a1aa-4031-8112-4e69e00e103b","Type":"ContainerStarted","Data":"fd6504f1c437db341ef21d40fe6d6e38b2f1128ccc774306e3635a4c7cb6e119"} Nov 02 14:07:37.542211 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:37.542157 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" podStartSLOduration=1.318555356 podStartE2EDuration="1.542135605s" totalImagesPullingTime="223.580249ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:37.539321522 +0000 UTC m=+622.723168858" watchObservedRunningTime="2026-11-02 14:07:37.542135605 +0000 UTC m=+622.725982943" Nov 02 14:07:37.827505 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:37.827475 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:39.092360 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:39.092313 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:39.108335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:39.108309 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:39.527354 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:39.527321 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nnfxn" Nov 02 14:07:39.643047 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:39.643004 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:07:39.643184 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:39.643099 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:07:39.643184 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:39.643156 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:07:40.531364 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:40.531322 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" event={"ID":"1fbf701c-a1aa-4031-8112-4e69e00e103b","Type":"ContainerStarted","Data":"b7001b065cc4bde710e1b057516dac2a3211edda9167ee7e9a39cfd765288ce6"} Nov 02 14:07:40.588473 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:40.588410 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" podStartSLOduration=1.7820975510000001 podStartE2EDuration="4.588392129s" totalImagesPullingTime="2.806294578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:40.587713547 +0000 UTC m=+625.771560909" watchObservedRunningTime="2026-11-02 14:07:40.588392129 +0000 UTC m=+625.772239468" Nov 02 14:07:41.492913 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:41.492882 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:41.494090 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:41.494061 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-8q5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 02 14:07:41.494165 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:41.494117 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" podUID="1fbf701c-a1aa-4031-8112-4e69e00e103b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 02 14:07:42.493209 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:42.493170 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-8q5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 02 14:07:42.493594 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:42.493226 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" podUID="1fbf701c-a1aa-4031-8112-4e69e00e103b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 02 14:07:42.894507 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:42.894421 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:42.923292 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:42.923256 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:43.493606 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:43.493574 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-8q5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 02 14:07:43.494013 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:43.493629 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" podUID="1fbf701c-a1aa-4031-8112-4e69e00e103b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 02 14:07:43.555437 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:43.555410 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-92h45" Nov 02 14:07:44.492870 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:44.492828 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-8q5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 02 14:07:44.493103 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:44.492883 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" podUID="1fbf701c-a1aa-4031-8112-4e69e00e103b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 02 14:07:45.493230 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:45.493197 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-8q5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 02 14:07:45.493602 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:45.493257 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" podUID="1fbf701c-a1aa-4031-8112-4e69e00e103b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 02 14:07:46.492723 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:46.492688 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-8q5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 02 14:07:46.492906 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:46.492752 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" podUID="1fbf701c-a1aa-4031-8112-4e69e00e103b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 02 14:07:46.827783 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:46.827688 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:46.840507 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:46.840473 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:47.576230 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:47.576199 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:07:47.923218 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:47.923128 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:47.923675 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:47.923605 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:47.924431 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:47.924406 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" Nov 02 14:07:52.980292 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:52.980258 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-64c547bcd4-s7w4p"] Nov 02 14:07:52.986794 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:52.986773 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64c547bcd4-s7w4p" Nov 02 14:07:53.005050 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:53.005011 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64c547bcd4-s7w4p"] Nov 02 14:07:53.043043 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:53.043004 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a9a48536-da8d-42b1-ac12-d5addcc2cadc-tls-cert\") pod \"authorino-64c547bcd4-s7w4p\" (UID: \"a9a48536-da8d-42b1-ac12-d5addcc2cadc\") " pod="kuadrant-system/authorino-64c547bcd4-s7w4p" Nov 02 14:07:53.043043 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:53.043040 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzjdm\" (UniqueName: \"kubernetes.io/projected/a9a48536-da8d-42b1-ac12-d5addcc2cadc-kube-api-access-rzjdm\") pod \"authorino-64c547bcd4-s7w4p\" (UID: \"a9a48536-da8d-42b1-ac12-d5addcc2cadc\") " pod="kuadrant-system/authorino-64c547bcd4-s7w4p" Nov 02 14:07:53.143796 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:53.143755 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a9a48536-da8d-42b1-ac12-d5addcc2cadc-tls-cert\") pod \"authorino-64c547bcd4-s7w4p\" (UID: \"a9a48536-da8d-42b1-ac12-d5addcc2cadc\") " pod="kuadrant-system/authorino-64c547bcd4-s7w4p" Nov 02 14:07:53.143796 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:53.143797 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzjdm\" (UniqueName: \"kubernetes.io/projected/a9a48536-da8d-42b1-ac12-d5addcc2cadc-kube-api-access-rzjdm\") pod \"authorino-64c547bcd4-s7w4p\" (UID: \"a9a48536-da8d-42b1-ac12-d5addcc2cadc\") " pod="kuadrant-system/authorino-64c547bcd4-s7w4p" Nov 02 14:07:53.146453 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:53.146425 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a9a48536-da8d-42b1-ac12-d5addcc2cadc-tls-cert\") pod \"authorino-64c547bcd4-s7w4p\" (UID: \"a9a48536-da8d-42b1-ac12-d5addcc2cadc\") " pod="kuadrant-system/authorino-64c547bcd4-s7w4p" Nov 02 14:07:53.216756 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:53.216726 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzjdm\" (UniqueName: \"kubernetes.io/projected/a9a48536-da8d-42b1-ac12-d5addcc2cadc-kube-api-access-rzjdm\") pod \"authorino-64c547bcd4-s7w4p\" (UID: \"a9a48536-da8d-42b1-ac12-d5addcc2cadc\") " pod="kuadrant-system/authorino-64c547bcd4-s7w4p" Nov 02 14:07:53.298832 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:53.298756 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64c547bcd4-s7w4p" Nov 02 14:07:53.597468 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:53.597442 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64c547bcd4-s7w4p"] Nov 02 14:07:53.599604 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:07:53.599580 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9a48536_da8d_42b1_ac12_d5addcc2cadc.slice/crio-0732d1e614dd18c602ed283d1e142b73ff29ef53e8ecf1bc3dac9f3032ebc993 WatchSource:0}: Error finding container 0732d1e614dd18c602ed283d1e142b73ff29ef53e8ecf1bc3dac9f3032ebc993: Status 404 returned error can't find the container with id 0732d1e614dd18c602ed283d1e142b73ff29ef53e8ecf1bc3dac9f3032ebc993 Nov 02 14:07:54.597174 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:54.597133 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64c547bcd4-s7w4p" event={"ID":"a9a48536-da8d-42b1-ac12-d5addcc2cadc","Type":"ContainerStarted","Data":"ef58575907ff8f2142195101b28855ef0d542ddf78a0b90200cf5c0f3723b30b"} Nov 02 14:07:54.597174 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:54.597179 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64c547bcd4-s7w4p" event={"ID":"a9a48536-da8d-42b1-ac12-d5addcc2cadc","Type":"ContainerStarted","Data":"0732d1e614dd18c602ed283d1e142b73ff29ef53e8ecf1bc3dac9f3032ebc993"} Nov 02 14:07:54.640907 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:54.640840 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-64c547bcd4-s7w4p" podStartSLOduration=2.12677137 podStartE2EDuration="2.640821177s" totalImagesPullingTime="514.049807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:54.639198103 +0000 UTC m=+639.823045460" watchObservedRunningTime="2026-11-02 14:07:54.640821177 +0000 UTC m=+639.824668519" Nov 02 14:07:54.684807 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:54.684771 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c75459655-fdx74"] Nov 02 14:07:54.685053 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:54.684982 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-c75459655-fdx74" podUID="60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a" containerName="authorino" containerID="cri-o://6cae6ac413e6a50efbfd35cf64d17bbc627e121dee95a8fafe7218361ed2ee98" gracePeriod=30 Nov 02 14:07:54.956706 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:54.956676 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c75459655-fdx74" Nov 02 14:07:55.063940 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:55.063900 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a-tls-cert\" (UniqueName: \"kubernetes.io/secret/60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a-tls-cert\") pod \"60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a\" (UID: \"60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a\") " Nov 02 14:07:55.064120 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:55.063989 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a-kube-api-access-9lsdb\" (UniqueName: \"kubernetes.io/projected/60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a-kube-api-access-9lsdb\") pod \"60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a\" (UID: \"60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a\") " Nov 02 14:07:55.066067 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:55.066034 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a-kube-api-access-9lsdb" pod "60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a" (UID: "60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a"). InnerVolumeSpecName "kube-api-access-9lsdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:07:55.075325 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:55.075292 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a-tls-cert" pod "60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a" (UID: "60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:07:55.164837 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:55.164754 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lsdb\" (UniqueName: \"kubernetes.io/projected/60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a-kube-api-access-9lsdb\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:07:55.164837 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:55.164784 4866 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a-tls-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:07:55.602936 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:55.602902 4866 generic.go:356] "Generic (PLEG): container finished" podID="60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a" containerID="6cae6ac413e6a50efbfd35cf64d17bbc627e121dee95a8fafe7218361ed2ee98" exitCode=0 Nov 02 14:07:55.603426 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:55.602978 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c75459655-fdx74" Nov 02 14:07:55.603426 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:55.602998 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c75459655-fdx74" event={"ID":"60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a","Type":"ContainerDied","Data":"6cae6ac413e6a50efbfd35cf64d17bbc627e121dee95a8fafe7218361ed2ee98"} Nov 02 14:07:55.603426 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:55.603048 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c75459655-fdx74" event={"ID":"60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a","Type":"ContainerDied","Data":"1770341ada5942f924a99450c7ed353af1bf0a6d67474c182447bfd4bbca5626"} Nov 02 14:07:55.603426 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:55.603072 4866 scope.go:122] "RemoveContainer" containerID="6cae6ac413e6a50efbfd35cf64d17bbc627e121dee95a8fafe7218361ed2ee98" Nov 02 14:07:55.613970 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:55.613919 4866 scope.go:122] "RemoveContainer" containerID="6cae6ac413e6a50efbfd35cf64d17bbc627e121dee95a8fafe7218361ed2ee98" Nov 02 14:07:55.614229 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:07:55.614210 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cae6ac413e6a50efbfd35cf64d17bbc627e121dee95a8fafe7218361ed2ee98\": container with ID starting with 6cae6ac413e6a50efbfd35cf64d17bbc627e121dee95a8fafe7218361ed2ee98 not found: ID does not exist" containerID="6cae6ac413e6a50efbfd35cf64d17bbc627e121dee95a8fafe7218361ed2ee98" Nov 02 14:07:55.614311 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:55.614243 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cae6ac413e6a50efbfd35cf64d17bbc627e121dee95a8fafe7218361ed2ee98"} err="failed to get container status \"6cae6ac413e6a50efbfd35cf64d17bbc627e121dee95a8fafe7218361ed2ee98\": rpc error: code = NotFound desc = could not find container \"6cae6ac413e6a50efbfd35cf64d17bbc627e121dee95a8fafe7218361ed2ee98\": container with ID starting with 6cae6ac413e6a50efbfd35cf64d17bbc627e121dee95a8fafe7218361ed2ee98 not found: ID does not exist" Nov 02 14:07:55.676090 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:55.676047 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c75459655-fdx74"] Nov 02 14:07:55.684935 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:55.684902 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c75459655-fdx74"] Nov 02 14:07:56.944775 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:07:56.944743 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a" path="/var/lib/kubelet/pods/60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a/volumes" Nov 02 14:08:06.008628 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:08:06.008593 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:08:06.275145 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.275069 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8"] Nov 02 14:08:06.275444 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.275432 4866 cpu_manager.go:414] "containerMap: removing container" podUID="60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a" containerName="authorino" Nov 02 14:08:06.275492 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.275445 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a" containerName="authorino" Nov 02 14:08:06.275546 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.275538 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="60fcc00b-b8c8-4eff-8cf3-ffb70b0fab3a" containerName="authorino" Nov 02 14:08:06.278927 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.278907 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.312879 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.312847 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8"] Nov 02 14:08:06.313068 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.312911 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk"] Nov 02 14:08:06.316241 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.316220 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.338752 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.338721 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk"] Nov 02 14:08:06.372223 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.372178 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd410bed-b4ef-467a-95f9-9b520abfd44d-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.372411 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.372289 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd410bed-b4ef-467a-95f9-9b520abfd44d-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.372411 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.372355 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.372411 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.372378 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.372411 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.372405 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.372623 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.372501 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd410bed-b4ef-467a-95f9-9b520abfd44d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.372623 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.372522 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.372623 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.372599 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.372775 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.372634 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd410bed-b4ef-467a-95f9-9b520abfd44d-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.372775 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.372687 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd410bed-b4ef-467a-95f9-9b520abfd44d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.372775 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.372707 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd410bed-b4ef-467a-95f9-9b520abfd44d-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.372775 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.372728 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggqhn\" (UniqueName: \"kubernetes.io/projected/cd410bed-b4ef-467a-95f9-9b520abfd44d-kube-api-access-ggqhn\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.372775 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.372767 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.373045 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.372788 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfjs6\" (UniqueName: \"kubernetes.io/projected/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-kube-api-access-lfjs6\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.373045 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.372806 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.373045 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.372824 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd410bed-b4ef-467a-95f9-9b520abfd44d-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.373045 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.372852 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.373045 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.372960 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd410bed-b4ef-467a-95f9-9b520abfd44d-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.473941 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.473902 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd410bed-b4ef-467a-95f9-9b520abfd44d-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.474139 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474055 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.474139 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474089 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.474248 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474199 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.474300 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474285 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd410bed-b4ef-467a-95f9-9b520abfd44d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.474352 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474313 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.474352 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474342 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd410bed-b4ef-467a-95f9-9b520abfd44d-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.474463 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474356 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.474463 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474420 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd410bed-b4ef-467a-95f9-9b520abfd44d-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.474463 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474456 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd410bed-b4ef-467a-95f9-9b520abfd44d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.474657 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474487 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd410bed-b4ef-467a-95f9-9b520abfd44d-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.474657 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474523 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggqhn\" (UniqueName: \"kubernetes.io/projected/cd410bed-b4ef-467a-95f9-9b520abfd44d-kube-api-access-ggqhn\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.474657 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474549 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.474657 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474580 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfjs6\" (UniqueName: \"kubernetes.io/projected/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-kube-api-access-lfjs6\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.474657 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474600 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.474657 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474612 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.474657 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474642 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd410bed-b4ef-467a-95f9-9b520abfd44d-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.475059 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474667 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.475059 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474704 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd410bed-b4ef-467a-95f9-9b520abfd44d-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.475059 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474739 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd410bed-b4ef-467a-95f9-9b520abfd44d-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.475059 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.474974 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.475059 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.475032 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd410bed-b4ef-467a-95f9-9b520abfd44d-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.475379 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.475229 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd410bed-b4ef-467a-95f9-9b520abfd44d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.475379 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.475311 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd410bed-b4ef-467a-95f9-9b520abfd44d-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.475488 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.475420 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd410bed-b4ef-467a-95f9-9b520abfd44d-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.475811 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.475786 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.475983 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.475964 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.476092 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.476065 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.477108 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.477087 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.477265 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.477244 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd410bed-b4ef-467a-95f9-9b520abfd44d-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.477380 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.477362 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd410bed-b4ef-467a-95f9-9b520abfd44d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.477937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.477917 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.497697 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.497676 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.498685 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.498665 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfjs6\" (UniqueName: \"kubernetes.io/projected/e3459850-2b2b-4dc0-b1ab-5ffbc87263a6-kube-api-access-lfjs6\") pod \"maas-default-gateway-openshift-default-55fbd65575-9pqlk\" (UID: \"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.500232 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.500214 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd410bed-b4ef-467a-95f9-9b520abfd44d-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.501184 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.501167 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggqhn\" (UniqueName: \"kubernetes.io/projected/cd410bed-b4ef-467a-95f9-9b520abfd44d-kube-api-access-ggqhn\") pod \"maas-default-gateway-openshift-default-55fbd65575-259d8\" (UID: \"cd410bed-b4ef-467a-95f9-9b520abfd44d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.591484 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.591405 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:06.629751 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.629712 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:06.919888 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.919866 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8"] Nov 02 14:08:06.922662 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:08:06.922639 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd410bed_b4ef_467a_95f9_9b520abfd44d.slice/crio-fe943d1cebd1520670a1d6335e8af92ffed730e7cc32537861bc17f61d1c137d WatchSource:0}: Error finding container fe943d1cebd1520670a1d6335e8af92ffed730e7cc32537861bc17f61d1c137d: Status 404 returned error can't find the container with id fe943d1cebd1520670a1d6335e8af92ffed730e7cc32537861bc17f61d1c137d Nov 02 14:08:06.924751 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.924719 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:08:06.924836 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.924783 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:08:06.924836 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.924816 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:08:06.964620 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.964596 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk"] Nov 02 14:08:06.966915 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:08:06.966890 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3459850_2b2b_4dc0_b1ab_5ffbc87263a6.slice/crio-f46e8d5275a3d6924a2b60560fe5025339f11abe474bc747e7748d185f8de487 WatchSource:0}: Error finding container f46e8d5275a3d6924a2b60560fe5025339f11abe474bc747e7748d185f8de487: Status 404 returned error can't find the container with id f46e8d5275a3d6924a2b60560fe5025339f11abe474bc747e7748d185f8de487 Nov 02 14:08:06.969211 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.969180 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:08:06.969313 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.969254 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:08:06.969313 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:06.969299 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:08:07.669414 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:07.669369 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" event={"ID":"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6","Type":"ContainerStarted","Data":"817a102cfbeca530454abda1f65101c0d217d75fd934a82f6d31310ee984c249"} Nov 02 14:08:07.669916 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:07.669430 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" event={"ID":"e3459850-2b2b-4dc0-b1ab-5ffbc87263a6","Type":"ContainerStarted","Data":"f46e8d5275a3d6924a2b60560fe5025339f11abe474bc747e7748d185f8de487"} Nov 02 14:08:07.671049 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:07.671026 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" event={"ID":"cd410bed-b4ef-467a-95f9-9b520abfd44d","Type":"ContainerStarted","Data":"a4840748e1adfd694b5a4d59af0f975e2a5a824cd33ece423b8839a175bf3880"} Nov 02 14:08:07.671155 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:07.671071 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" event={"ID":"cd410bed-b4ef-467a-95f9-9b520abfd44d","Type":"ContainerStarted","Data":"fe943d1cebd1520670a1d6335e8af92ffed730e7cc32537861bc17f61d1c137d"} Nov 02 14:08:07.732973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:07.732818 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" podStartSLOduration=1.7327985479999999 podStartE2EDuration="1.732798548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:07.731451021 +0000 UTC m=+652.915298373" watchObservedRunningTime="2026-11-02 14:08:07.732798548 +0000 UTC m=+652.916645886" Nov 02 14:08:07.779894 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:07.779839 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" podStartSLOduration=1.779825342 podStartE2EDuration="1.779825342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:07.779603938 +0000 UTC m=+652.963451274" watchObservedRunningTime="2026-11-02 14:08:07.779825342 +0000 UTC m=+652.963672678" Nov 02 14:08:08.592748 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:08.592714 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:08.594140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:08.594117 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-259d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 02 14:08:08.594197 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:08.594168 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" podUID="cd410bed-b4ef-467a-95f9-9b520abfd44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 02 14:08:08.630791 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:08.630446 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:08.635978 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:08.635285 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-9pqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 02 14:08:08.635978 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:08.635353 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" podUID="e3459850-2b2b-4dc0-b1ab-5ffbc87263a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 02 14:08:09.592244 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:09.592214 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-259d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 02 14:08:09.592705 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:09.592680 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" podUID="cd410bed-b4ef-467a-95f9-9b520abfd44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 02 14:08:09.630971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:09.630500 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-9pqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 02 14:08:09.630971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:09.630579 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" podUID="e3459850-2b2b-4dc0-b1ab-5ffbc87263a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 02 14:08:10.592122 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:10.592086 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-259d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 02 14:08:10.592339 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:10.592145 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" podUID="cd410bed-b4ef-467a-95f9-9b520abfd44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 02 14:08:10.633983 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:10.630391 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-9pqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 02 14:08:10.633983 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:10.630457 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" podUID="e3459850-2b2b-4dc0-b1ab-5ffbc87263a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 02 14:08:11.592372 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:11.592333 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-259d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 02 14:08:11.592770 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:11.592396 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" podUID="cd410bed-b4ef-467a-95f9-9b520abfd44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 02 14:08:11.630971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:11.630389 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-9pqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 02 14:08:11.630971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:11.630466 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" podUID="e3459850-2b2b-4dc0-b1ab-5ffbc87263a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 02 14:08:12.591892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:12.591859 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-259d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 02 14:08:12.592114 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:12.591912 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" podUID="cd410bed-b4ef-467a-95f9-9b520abfd44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 02 14:08:12.632195 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:12.632160 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-9pqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 02 14:08:12.632896 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:12.632767 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" podUID="e3459850-2b2b-4dc0-b1ab-5ffbc87263a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 02 14:08:13.592542 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:13.592505 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-259d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 02 14:08:13.592806 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:13.592557 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" podUID="cd410bed-b4ef-467a-95f9-9b520abfd44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 02 14:08:13.630609 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:13.630569 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-9pqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 02 14:08:13.630774 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:13.630636 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" podUID="e3459850-2b2b-4dc0-b1ab-5ffbc87263a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 02 14:08:15.303172 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:15.303095 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:15.304030 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:15.304006 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:15.305067 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:15.305045 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" Nov 02 14:08:15.380064 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:15.380025 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:15.380435 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:15.380403 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:15.381057 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:15.381038 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" Nov 02 14:08:36.007831 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:08:36.007748 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:08:52.369301 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.369266 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4"] Nov 02 14:08:52.374718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.374696 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.389610 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.389586 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-555d1e-openshift-default-dockercfg-s5d5d\"" Nov 02 14:08:52.403240 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.403214 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4"] Nov 02 14:08:52.514278 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.514249 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q"] Nov 02 14:08:52.517833 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.517809 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.535254 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.535226 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-4be645-openshift-default-dockercfg-2ds72\"" Nov 02 14:08:52.543150 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.543120 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q"] Nov 02 14:08:52.563149 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.563117 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.563314 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.563230 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.563371 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.563335 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.563433 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.563418 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.563489 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.563443 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n29mg\" (UniqueName: \"kubernetes.io/projected/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-kube-api-access-n29mg\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.563564 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.563546 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.563635 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.563620 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.563720 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.563700 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.563847 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.563738 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.664562 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.664485 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.664562 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.664528 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.664562 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.664552 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.664801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.664568 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cprfx\" (UniqueName: \"kubernetes.io/projected/38537e17-d921-43b8-aa90-15a82b39342b-kube-api-access-cprfx\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.664801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.664663 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n29mg\" (UniqueName: \"kubernetes.io/projected/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-kube-api-access-n29mg\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.664801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.664703 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.664801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.664727 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.664801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.664758 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.665085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.664855 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.665085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.664923 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.665085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.664930 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.665085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.665018 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38537e17-d921-43b8-aa90-15a82b39342b-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.665085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.665053 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.665085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.665055 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.665384 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.665128 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.665384 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.665150 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.665384 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.665185 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.665384 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.665236 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38537e17-d921-43b8-aa90-15a82b39342b-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.665384 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.665243 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.665384 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.665261 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.665384 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.665317 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38537e17-d921-43b8-aa90-15a82b39342b-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.665615 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.665592 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.665665 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.665648 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.667052 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.667034 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.668088 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.668072 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.696717 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.696667 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n29mg\" (UniqueName: \"kubernetes.io/projected/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-kube-api-access-n29mg\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.699313 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.699282 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:52.766248 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.766198 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.766248 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.766252 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cprfx\" (UniqueName: \"kubernetes.io/projected/38537e17-d921-43b8-aa90-15a82b39342b-kube-api-access-cprfx\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.766496 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.766437 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.766496 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.766493 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.766640 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.766529 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.766640 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.766580 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38537e17-d921-43b8-aa90-15a82b39342b-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.766640 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.766634 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.766809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.766668 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.766809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.766715 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38537e17-d921-43b8-aa90-15a82b39342b-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.766809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.766761 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38537e17-d921-43b8-aa90-15a82b39342b-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.766809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.766764 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.767065 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.766819 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.767065 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.766911 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.767436 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.767418 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38537e17-d921-43b8-aa90-15a82b39342b-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.768874 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.768847 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.769269 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.769248 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38537e17-d921-43b8-aa90-15a82b39342b-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.789534 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.789495 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cprfx\" (UniqueName: \"kubernetes.io/projected/38537e17-d921-43b8-aa90-15a82b39342b-kube-api-access-cprfx\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.790390 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.790372 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38537e17-d921-43b8-aa90-15a82b39342b-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.836128 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.836093 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:52.987803 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:52.987763 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:53.184082 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:53.184017 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q"] Nov 02 14:08:53.186593 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:08:53.186564 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38537e17_d921_43b8_aa90_15a82b39342b.slice/crio-7a13919aa1638de25109329396c2aa8b3b8c01ab80f6288fb71b536bc0ca0130 WatchSource:0}: Error finding container 7a13919aa1638de25109329396c2aa8b3b8c01ab80f6288fb71b536bc0ca0130: Status 404 returned error can't find the container with id 7a13919aa1638de25109329396c2aa8b3b8c01ab80f6288fb71b536bc0ca0130 Nov 02 14:08:53.189183 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:53.189154 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:08:53.189244 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:53.189215 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:08:53.189283 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:53.189246 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:08:53.339652 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:53.339617 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4"] Nov 02 14:08:53.343152 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:08:53.343122 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ebf7e9_7f0f_4489_893d_d2ad6efc1910.slice/crio-d9ceb0544315a10f31cca5bde5f9db7fd738761d1202f65f79561e036acfc146 WatchSource:0}: Error finding container d9ceb0544315a10f31cca5bde5f9db7fd738761d1202f65f79561e036acfc146: Status 404 returned error can't find the container with id d9ceb0544315a10f31cca5bde5f9db7fd738761d1202f65f79561e036acfc146 Nov 02 14:08:53.345643 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:53.345603 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:08:53.345724 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:53.345687 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:08:53.345784 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:53.345731 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:08:53.900639 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:53.900593 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" event={"ID":"38537e17-d921-43b8-aa90-15a82b39342b","Type":"ContainerStarted","Data":"5e15c0dd5dbc9280f3bae13a30f10045b2003fa01649f000771874e730d897c9"} Nov 02 14:08:53.901158 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:53.900667 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" event={"ID":"38537e17-d921-43b8-aa90-15a82b39342b","Type":"ContainerStarted","Data":"7a13919aa1638de25109329396c2aa8b3b8c01ab80f6288fb71b536bc0ca0130"} Nov 02 14:08:53.902289 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:53.902260 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" event={"ID":"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910","Type":"ContainerStarted","Data":"f57d6ee3c1381a21cf89a46a89b45153435a3f9c8285aeeaf8dfc37387b55e8b"} Nov 02 14:08:53.902408 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:53.902321 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" event={"ID":"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910","Type":"ContainerStarted","Data":"d9ceb0544315a10f31cca5bde5f9db7fd738761d1202f65f79561e036acfc146"} Nov 02 14:08:53.971443 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:53.971383 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" podStartSLOduration=1.97136801 podStartE2EDuration="1.97136801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:53.959218139 +0000 UTC m=+699.143065479" watchObservedRunningTime="2026-11-02 14:08:53.97136801 +0000 UTC m=+699.155215346" Nov 02 14:08:53.987957 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:53.987928 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:54.013169 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:54.013107 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" podStartSLOduration=2.013089339 podStartE2EDuration="2.013089339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:54.01069368 +0000 UTC m=+699.194541017" watchObservedRunningTime="2026-11-02 14:08:54.013089339 +0000 UTC m=+699.196936677" Nov 02 14:08:54.836721 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:54.836686 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:54.841929 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:54.841900 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:54.907700 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:54.907672 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:54.908401 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:54.908382 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:08:54.994041 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:54.994008 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:55.912957 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:55.912924 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:55.913790 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:55.913770 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:08:57.664497 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:57.664466 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf"] Nov 02 14:08:57.668095 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:57.668073 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" Nov 02 14:08:57.686619 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:57.686594 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-c0c28f-serving-cert\"" Nov 02 14:08:57.695759 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:57.695692 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf"] Nov 02 14:08:57.764432 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:57.764400 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-metrics-certs\") pod \"maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf\" (UID: \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" Nov 02 14:08:57.764609 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:57.764462 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf\" (UID: \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" Nov 02 14:08:57.764609 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:57.764587 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q7d7\" (UniqueName: \"kubernetes.io/projected/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-kube-api-access-2q7d7\") pod \"maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf\" (UID: \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" Nov 02 14:08:57.764686 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:57.764633 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf\" (UID: \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" Nov 02 14:08:57.865501 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:57.865466 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf\" (UID: \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" Nov 02 14:08:57.865676 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:57.865553 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q7d7\" (UniqueName: \"kubernetes.io/projected/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-kube-api-access-2q7d7\") pod \"maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf\" (UID: \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" Nov 02 14:08:57.865676 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:57.865587 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf\" (UID: \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" Nov 02 14:08:57.865676 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:57.865654 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-metrics-certs\") pod \"maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf\" (UID: \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" Nov 02 14:08:57.866523 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:57.866493 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf\" (UID: \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" Nov 02 14:08:57.868054 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:57.868028 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf\" (UID: \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" Nov 02 14:08:57.868054 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:57.868048 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-metrics-certs\") pod \"maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf\" (UID: \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" Nov 02 14:08:57.898151 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:57.898123 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q7d7\" (UniqueName: \"kubernetes.io/projected/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-kube-api-access-2q7d7\") pod \"maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf\" (UID: \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" Nov 02 14:08:57.982141 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:57.982103 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" Nov 02 14:08:58.313136 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:58.313051 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf"] Nov 02 14:08:58.316089 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:08:58.316064 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3dcadd1_584c_4dee_b0c0_402f8b4175c5.slice/crio-68043d61c1bb7cfba261b2ac95374171fdb2627b9f5040a3fedde1b0ff16ee35 WatchSource:0}: Error finding container 68043d61c1bb7cfba261b2ac95374171fdb2627b9f5040a3fedde1b0ff16ee35: Status 404 returned error can't find the container with id 68043d61c1bb7cfba261b2ac95374171fdb2627b9f5040a3fedde1b0ff16ee35 Nov 02 14:08:58.927668 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:58.927617 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" event={"ID":"a3dcadd1-584c-4dee-b0c0-402f8b4175c5","Type":"ContainerStarted","Data":"d933858ddebde1fe38a03b1affd300a87ab601c04a154138eb6be74f17650765"} Nov 02 14:08:58.927668 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:58.927671 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" event={"ID":"a3dcadd1-584c-4dee-b0c0-402f8b4175c5","Type":"ContainerStarted","Data":"68043d61c1bb7cfba261b2ac95374171fdb2627b9f5040a3fedde1b0ff16ee35"} Nov 02 14:08:58.928264 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:58.927699 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" Nov 02 14:08:58.974844 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:08:58.974495 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" podStartSLOduration=1.532360022 podStartE2EDuration="1.974477819s" totalImagesPullingTime="442.117797ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:58.972052003 +0000 UTC m=+704.155899340" watchObservedRunningTime="2026-11-02 14:08:58.974477819 +0000 UTC m=+704.158325158" Nov 02 14:09:00.770087 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:00.770046 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g"] Nov 02 14:09:00.774702 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:00.774668 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" Nov 02 14:09:00.791287 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:00.791257 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g"] Nov 02 14:09:00.795157 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:00.795130 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px9p5\" (UniqueName: \"kubernetes.io/projected/e7cb0799-8ad5-431f-850d-fbfaa881f9e5-kube-api-access-px9p5\") pod \"payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g\" (UID: \"e7cb0799-8ad5-431f-850d-fbfaa881f9e5\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" Nov 02 14:09:00.795319 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:00.795296 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7cb0799-8ad5-431f-850d-fbfaa881f9e5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g\" (UID: \"e7cb0799-8ad5-431f-850d-fbfaa881f9e5\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" Nov 02 14:09:00.811484 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:00.811456 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-555d1e\"" Nov 02 14:09:00.811632 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:00.811568 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-555d1e-dockercfg-9zkr7\"" Nov 02 14:09:00.896554 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:00.896521 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7cb0799-8ad5-431f-850d-fbfaa881f9e5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g\" (UID: \"e7cb0799-8ad5-431f-850d-fbfaa881f9e5\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" Nov 02 14:09:00.896725 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:00.896604 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px9p5\" (UniqueName: \"kubernetes.io/projected/e7cb0799-8ad5-431f-850d-fbfaa881f9e5-kube-api-access-px9p5\") pod \"payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g\" (UID: \"e7cb0799-8ad5-431f-850d-fbfaa881f9e5\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" Nov 02 14:09:00.897304 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:00.897284 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7cb0799-8ad5-431f-850d-fbfaa881f9e5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g\" (UID: \"e7cb0799-8ad5-431f-850d-fbfaa881f9e5\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" Nov 02 14:09:00.920030 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:00.920003 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px9p5\" (UniqueName: \"kubernetes.io/projected/e7cb0799-8ad5-431f-850d-fbfaa881f9e5-kube-api-access-px9p5\") pod \"payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g\" (UID: \"e7cb0799-8ad5-431f-850d-fbfaa881f9e5\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" Nov 02 14:09:01.092218 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:01.092134 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" Nov 02 14:09:01.389548 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:01.389521 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g"] Nov 02 14:09:01.391418 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:09:01.391381 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7cb0799_8ad5_431f_850d_fbfaa881f9e5.slice/crio-a810dc9a134e422548c9d80dea1ecb6fb78739140e01b993e27adfd09b986c80 WatchSource:0}: Error finding container a810dc9a134e422548c9d80dea1ecb6fb78739140e01b993e27adfd09b986c80: Status 404 returned error can't find the container with id a810dc9a134e422548c9d80dea1ecb6fb78739140e01b993e27adfd09b986c80 Nov 02 14:09:01.944060 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:01.944024 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" event={"ID":"e7cb0799-8ad5-431f-850d-fbfaa881f9e5","Type":"ContainerStarted","Data":"4836afc1eaaf69f6c3f959d78019024f1ef31b9a7d774ad54b69ce9fc4184313"} Nov 02 14:09:01.944441 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:01.944074 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" event={"ID":"e7cb0799-8ad5-431f-850d-fbfaa881f9e5","Type":"ContainerStarted","Data":"a810dc9a134e422548c9d80dea1ecb6fb78739140e01b993e27adfd09b986c80"} Nov 02 14:09:01.944441 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:01.944113 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" Nov 02 14:09:01.986633 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:01.986564 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" podStartSLOduration=1.9865464 podStartE2EDuration="1.9865464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:01.984292121 +0000 UTC m=+707.168139458" watchObservedRunningTime="2026-11-02 14:09:01.9865464 +0000 UTC m=+707.170393738" Nov 02 14:09:03.407303 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:03.407265 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr"] Nov 02 14:09:03.411771 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:03.411745 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:09:03.442030 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:03.436299 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-4be645-serving-cert\"" Nov 02 14:09:03.446342 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:03.446313 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr"] Nov 02 14:09:03.523664 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:03.523626 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/861a94e0-defc-4d04-b11d-ecbdc66002b2-maas-api-tls\") pod \"maas-api-e2e-worker-w2-4be645-6cd878986-djglr\" (UID: \"861a94e0-defc-4d04-b11d-ecbdc66002b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:09:03.523808 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:03.523668 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/861a94e0-defc-4d04-b11d-ecbdc66002b2-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-4be645-6cd878986-djglr\" (UID: \"861a94e0-defc-4d04-b11d-ecbdc66002b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:09:03.523808 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:03.523788 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/861a94e0-defc-4d04-b11d-ecbdc66002b2-metrics-certs\") pod \"maas-api-e2e-worker-w2-4be645-6cd878986-djglr\" (UID: \"861a94e0-defc-4d04-b11d-ecbdc66002b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:09:03.523903 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:03.523814 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l9pf\" (UniqueName: \"kubernetes.io/projected/861a94e0-defc-4d04-b11d-ecbdc66002b2-kube-api-access-7l9pf\") pod \"maas-api-e2e-worker-w2-4be645-6cd878986-djglr\" (UID: \"861a94e0-defc-4d04-b11d-ecbdc66002b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:09:03.624767 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:03.624728 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/861a94e0-defc-4d04-b11d-ecbdc66002b2-maas-api-tls\") pod \"maas-api-e2e-worker-w2-4be645-6cd878986-djglr\" (UID: \"861a94e0-defc-4d04-b11d-ecbdc66002b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:09:03.624767 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:03.624767 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/861a94e0-defc-4d04-b11d-ecbdc66002b2-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-4be645-6cd878986-djglr\" (UID: \"861a94e0-defc-4d04-b11d-ecbdc66002b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:09:03.624986 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:03.624820 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/861a94e0-defc-4d04-b11d-ecbdc66002b2-metrics-certs\") pod \"maas-api-e2e-worker-w2-4be645-6cd878986-djglr\" (UID: \"861a94e0-defc-4d04-b11d-ecbdc66002b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:09:03.624986 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:03.624848 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l9pf\" (UniqueName: \"kubernetes.io/projected/861a94e0-defc-4d04-b11d-ecbdc66002b2-kube-api-access-7l9pf\") pod \"maas-api-e2e-worker-w2-4be645-6cd878986-djglr\" (UID: \"861a94e0-defc-4d04-b11d-ecbdc66002b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:09:03.624986 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:09:03.624882 4866 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-serving-cert: secret "maas-api-e2e-worker-w2-4be645-serving-cert" not found Nov 02 14:09:03.624986 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:09:03.624984 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/861a94e0-defc-4d04-b11d-ecbdc66002b2-maas-api-tls podName:861a94e0-defc-4d04-b11d-ecbdc66002b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:09:04.124938318 +0000 UTC m=+709.308785652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/861a94e0-defc-4d04-b11d-ecbdc66002b2-maas-api-tls") pod "maas-api-e2e-worker-w2-4be645-6cd878986-djglr" (UID: "861a94e0-defc-4d04-b11d-ecbdc66002b2") : secret "maas-api-e2e-worker-w2-4be645-serving-cert" not found Nov 02 14:09:03.625499 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:03.625473 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/861a94e0-defc-4d04-b11d-ecbdc66002b2-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-4be645-6cd878986-djglr\" (UID: \"861a94e0-defc-4d04-b11d-ecbdc66002b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:09:03.627250 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:03.627226 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/861a94e0-defc-4d04-b11d-ecbdc66002b2-metrics-certs\") pod \"maas-api-e2e-worker-w2-4be645-6cd878986-djglr\" (UID: \"861a94e0-defc-4d04-b11d-ecbdc66002b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:09:03.651120 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:03.651090 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l9pf\" (UniqueName: \"kubernetes.io/projected/861a94e0-defc-4d04-b11d-ecbdc66002b2-kube-api-access-7l9pf\") pod \"maas-api-e2e-worker-w2-4be645-6cd878986-djglr\" (UID: \"861a94e0-defc-4d04-b11d-ecbdc66002b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:09:04.129448 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:04.129407 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/861a94e0-defc-4d04-b11d-ecbdc66002b2-maas-api-tls\") pod \"maas-api-e2e-worker-w2-4be645-6cd878986-djglr\" (UID: \"861a94e0-defc-4d04-b11d-ecbdc66002b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:09:04.131843 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:04.131813 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/861a94e0-defc-4d04-b11d-ecbdc66002b2-maas-api-tls\") pod \"maas-api-e2e-worker-w2-4be645-6cd878986-djglr\" (UID: \"861a94e0-defc-4d04-b11d-ecbdc66002b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:09:04.325025 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:04.324993 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:09:04.631299 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:04.631272 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr"] Nov 02 14:09:04.634401 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:09:04.634373 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861a94e0_defc_4d04_b11d_ecbdc66002b2.slice/crio-542e0f8093cf0d76f73039103aa688086597157c699f7b66caec63be0aa81546 WatchSource:0}: Error finding container 542e0f8093cf0d76f73039103aa688086597157c699f7b66caec63be0aa81546: Status 404 returned error can't find the container with id 542e0f8093cf0d76f73039103aa688086597157c699f7b66caec63be0aa81546 Nov 02 14:09:04.945195 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:04.945170 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" Nov 02 14:09:04.957805 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:04.957777 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" event={"ID":"861a94e0-defc-4d04-b11d-ecbdc66002b2","Type":"ContainerStarted","Data":"542e0f8093cf0d76f73039103aa688086597157c699f7b66caec63be0aa81546"} Nov 02 14:09:05.297894 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:05.297858 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh"] Nov 02 14:09:05.301520 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:05.301501 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" Nov 02 14:09:05.317522 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:05.317500 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-555d1e-serving-cert\"" Nov 02 14:09:05.328435 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:05.328406 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh"] Nov 02 14:09:05.341838 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:05.341806 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1725bed-2118-40d7-bfa7-88cf6ca87354-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh\" (UID: \"b1725bed-2118-40d7-bfa7-88cf6ca87354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" Nov 02 14:09:05.342008 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:05.341871 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb76q\" (UniqueName: \"kubernetes.io/projected/b1725bed-2118-40d7-bfa7-88cf6ca87354-kube-api-access-jb76q\") pod \"maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh\" (UID: \"b1725bed-2118-40d7-bfa7-88cf6ca87354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" Nov 02 14:09:05.342080 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:05.342002 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1725bed-2118-40d7-bfa7-88cf6ca87354-metrics-certs\") pod \"maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh\" (UID: \"b1725bed-2118-40d7-bfa7-88cf6ca87354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" Nov 02 14:09:05.342080 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:05.342062 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1725bed-2118-40d7-bfa7-88cf6ca87354-maas-api-tls\") pod \"maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh\" (UID: \"b1725bed-2118-40d7-bfa7-88cf6ca87354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" Nov 02 14:09:05.443196 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:05.443162 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1725bed-2118-40d7-bfa7-88cf6ca87354-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh\" (UID: \"b1725bed-2118-40d7-bfa7-88cf6ca87354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" Nov 02 14:09:05.443341 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:05.443224 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb76q\" (UniqueName: \"kubernetes.io/projected/b1725bed-2118-40d7-bfa7-88cf6ca87354-kube-api-access-jb76q\") pod \"maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh\" (UID: \"b1725bed-2118-40d7-bfa7-88cf6ca87354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" Nov 02 14:09:05.443341 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:05.443247 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1725bed-2118-40d7-bfa7-88cf6ca87354-metrics-certs\") pod \"maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh\" (UID: \"b1725bed-2118-40d7-bfa7-88cf6ca87354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" Nov 02 14:09:05.443341 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:05.443272 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1725bed-2118-40d7-bfa7-88cf6ca87354-maas-api-tls\") pod \"maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh\" (UID: \"b1725bed-2118-40d7-bfa7-88cf6ca87354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" Nov 02 14:09:05.443462 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:09:05.443399 4866 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-serving-cert: secret "maas-api-e2e-worker-w0-555d1e-serving-cert" not found Nov 02 14:09:05.443462 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:09:05.443460 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1725bed-2118-40d7-bfa7-88cf6ca87354-maas-api-tls podName:b1725bed-2118-40d7-bfa7-88cf6ca87354 nodeName:}" failed. No retries permitted until 2026-11-02 14:09:05.943442063 +0000 UTC m=+711.127289382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1725bed-2118-40d7-bfa7-88cf6ca87354-maas-api-tls") pod "maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" (UID: "b1725bed-2118-40d7-bfa7-88cf6ca87354") : secret "maas-api-e2e-worker-w0-555d1e-serving-cert" not found Nov 02 14:09:05.443901 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:05.443872 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1725bed-2118-40d7-bfa7-88cf6ca87354-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh\" (UID: \"b1725bed-2118-40d7-bfa7-88cf6ca87354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" Nov 02 14:09:05.445960 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:05.445921 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1725bed-2118-40d7-bfa7-88cf6ca87354-metrics-certs\") pod \"maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh\" (UID: \"b1725bed-2118-40d7-bfa7-88cf6ca87354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" Nov 02 14:09:05.472160 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:05.472129 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb76q\" (UniqueName: \"kubernetes.io/projected/b1725bed-2118-40d7-bfa7-88cf6ca87354-kube-api-access-jb76q\") pod \"maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh\" (UID: \"b1725bed-2118-40d7-bfa7-88cf6ca87354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" Nov 02 14:09:05.947262 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:05.947230 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1725bed-2118-40d7-bfa7-88cf6ca87354-maas-api-tls\") pod \"maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh\" (UID: \"b1725bed-2118-40d7-bfa7-88cf6ca87354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" Nov 02 14:09:05.949626 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:05.949598 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1725bed-2118-40d7-bfa7-88cf6ca87354-maas-api-tls\") pod \"maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh\" (UID: \"b1725bed-2118-40d7-bfa7-88cf6ca87354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" Nov 02 14:09:05.964421 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:05.964383 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" event={"ID":"861a94e0-defc-4d04-b11d-ecbdc66002b2","Type":"ContainerStarted","Data":"fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991"} Nov 02 14:09:05.964545 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:05.964460 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:09:06.009147 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:09:06.009104 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:09:06.010057 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:06.010003 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" podStartSLOduration=2.723257297 podStartE2EDuration="3.009990147s" totalImagesPullingTime="286.73285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:06.008804851 +0000 UTC m=+711.192652189" watchObservedRunningTime="2026-11-02 14:09:06.009990147 +0000 UTC m=+711.193837481" Nov 02 14:09:06.213977 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:06.213871 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" Nov 02 14:09:06.514959 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:06.514927 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh"] Nov 02 14:09:06.517021 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:09:06.516994 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1725bed_2118_40d7_bfa7_88cf6ca87354.slice/crio-e6ce0582520cdea96b9c3cebe31009feee18f941fae31570d6d568c2822dc258 WatchSource:0}: Error finding container e6ce0582520cdea96b9c3cebe31009feee18f941fae31570d6d568c2822dc258: Status 404 returned error can't find the container with id e6ce0582520cdea96b9c3cebe31009feee18f941fae31570d6d568c2822dc258 Nov 02 14:09:06.970672 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:06.970635 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" event={"ID":"b1725bed-2118-40d7-bfa7-88cf6ca87354","Type":"ContainerStarted","Data":"ce02e2cd0972810846e4211d758320f49a0a2105618645e7517501723319204e"} Nov 02 14:09:06.971133 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:06.970781 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" Nov 02 14:09:06.971133 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:06.970803 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" event={"ID":"b1725bed-2118-40d7-bfa7-88cf6ca87354","Type":"ContainerStarted","Data":"e6ce0582520cdea96b9c3cebe31009feee18f941fae31570d6d568c2822dc258"} Nov 02 14:09:07.014751 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:07.014698 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" podStartSLOduration=1.729555764 podStartE2EDuration="2.014684054s" totalImagesPullingTime="285.12829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:07.012152921 +0000 UTC m=+712.196000257" watchObservedRunningTime="2026-11-02 14:09:07.014684054 +0000 UTC m=+712.198531392" Nov 02 14:09:11.972518 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:11.972480 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" podUID="861a94e0-defc-4d04-b11d-ecbdc66002b2" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.52:8443/health\": dial tcp 10.132.0.52:8443: connect: connection refused" Nov 02 14:09:12.984973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:12.969684 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" Nov 02 14:09:12.995804 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:12.995483 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" Nov 02 14:09:16.149690 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:16.149655 4866 generic.go:356] "Generic (PLEG): container finished" podID="861a94e0-defc-4d04-b11d-ecbdc66002b2" containerID="fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991" exitCode=1 Nov 02 14:09:16.150078 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:16.149742 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" event={"ID":"861a94e0-defc-4d04-b11d-ecbdc66002b2","Type":"ContainerDied","Data":"fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991"} Nov 02 14:09:16.150285 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:16.150266 4866 scope.go:122] "RemoveContainer" containerID="fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991" Nov 02 14:09:17.034202 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:09:17.034154 4866 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861a94e0_defc_4d04_b11d_ecbdc66002b2.slice/crio-conmon-fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991.scope\": RecentStats: unable to find data in memory cache]" Nov 02 14:09:17.157216 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:17.157172 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" event={"ID":"861a94e0-defc-4d04-b11d-ecbdc66002b2","Type":"ContainerStarted","Data":"2a8bc16aa21af53b3845151b1aeb18e37950195431a31212ade96927b435a4d0"} Nov 02 14:09:17.157649 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:17.157424 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:09:17.994238 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:09:17.994201 4866 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861a94e0_defc_4d04_b11d_ecbdc66002b2.slice/crio-conmon-fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991.scope\": RecentStats: unable to find data in memory cache]" Nov 02 14:09:18.446068 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.446033 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m"] Nov 02 14:09:18.450922 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.450894 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:18.470224 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.470192 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m"] Nov 02 14:09:18.479586 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.479557 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c0c28f\"/\"kube-root-ca.crt\"" Nov 02 14:09:18.479739 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.479716 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c0c28f\"/\"openshift-service-ca.crt\"" Nov 02 14:09:18.482835 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.482817 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c0c28f\"/\"default-dockercfg-hfg2t\"" Nov 02 14:09:18.483116 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.483098 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c0c28f\"/\"simulator-w4-c0c28f-kserve-self-signed-certs\"" Nov 02 14:09:18.646609 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.646585 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-tmp-dir\") pod \"simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:18.646822 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.646624 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9753ddaa-3899-46ea-952f-5f03d76037c6-tls-certs\") pod \"simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:18.646822 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.646646 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-model-cache\") pod \"simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:18.646822 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.646747 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdn8d\" (UniqueName: \"kubernetes.io/projected/9753ddaa-3899-46ea-952f-5f03d76037c6-kube-api-access-bdn8d\") pod \"simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:18.646982 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.646842 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-dshm\") pod \"simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:18.646982 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.646914 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-home\") pod \"simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:18.748355 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.748277 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-home\") pod \"simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:18.748355 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.748333 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-tmp-dir\") pod \"simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:18.748587 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.748377 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9753ddaa-3899-46ea-952f-5f03d76037c6-tls-certs\") pod \"simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:18.748587 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.748400 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-model-cache\") pod \"simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:18.748587 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.748448 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdn8d\" (UniqueName: \"kubernetes.io/projected/9753ddaa-3899-46ea-952f-5f03d76037c6-kube-api-access-bdn8d\") pod \"simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:18.748587 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.748507 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-dshm\") pod \"simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:18.748782 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.748711 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-home\") pod \"simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:18.748782 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.748773 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-tmp-dir\") pod \"simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:18.749188 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.749159 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-model-cache\") pod \"simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:18.750838 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.750806 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-dshm\") pod \"simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:18.750942 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.750815 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9753ddaa-3899-46ea-952f-5f03d76037c6-tls-certs\") pod \"simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:18.773107 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:18.773083 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdn8d\" (UniqueName: \"kubernetes.io/projected/9753ddaa-3899-46ea-952f-5f03d76037c6-kube-api-access-bdn8d\") pod \"simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:19.066743 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:19.066652 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:19.377853 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:19.377822 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m"] Nov 02 14:09:19.378934 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:09:19.378905 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9753ddaa_3899_46ea_952f_5f03d76037c6.slice/crio-7ba3f58714731f68ee81bac359e0ffd71dc983650030f4313c1d3b754cc84d50 WatchSource:0}: Error finding container 7ba3f58714731f68ee81bac359e0ffd71dc983650030f4313c1d3b754cc84d50: Status 404 returned error can't find the container with id 7ba3f58714731f68ee81bac359e0ffd71dc983650030f4313c1d3b754cc84d50 Nov 02 14:09:20.177031 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:20.176915 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" event={"ID":"9753ddaa-3899-46ea-952f-5f03d76037c6","Type":"ContainerStarted","Data":"e3f2beeb5ba2f2b69cee8f37a17604b46aba36ab4ba45c78a293f2fc46204c4e"} Nov 02 14:09:20.177031 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:20.176995 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" event={"ID":"9753ddaa-3899-46ea-952f-5f03d76037c6","Type":"ContainerStarted","Data":"7ba3f58714731f68ee81bac359e0ffd71dc983650030f4313c1d3b754cc84d50"} Nov 02 14:09:20.178561 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:20.178537 4866 generic.go:356] "Generic (PLEG): container finished" podID="38537e17-d921-43b8-aa90-15a82b39342b" containerID="5e15c0dd5dbc9280f3bae13a30f10045b2003fa01649f000771874e730d897c9" exitCode=0 Nov 02 14:09:20.178666 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:20.178572 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" event={"ID":"38537e17-d921-43b8-aa90-15a82b39342b","Type":"ContainerDied","Data":"5e15c0dd5dbc9280f3bae13a30f10045b2003fa01649f000771874e730d897c9"} Nov 02 14:09:20.178886 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:20.178874 4866 scope.go:122] "RemoveContainer" containerID="5e15c0dd5dbc9280f3bae13a30f10045b2003fa01649f000771874e730d897c9" Nov 02 14:09:20.180176 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:20.180151 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:09:20.180265 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:20.180226 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:09:20.180304 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:20.180263 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:09:20.220045 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:20.219809 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" podStartSLOduration=2.219795001 podStartE2EDuration="2.219795001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:20.218198173 +0000 UTC m=+725.402045510" watchObservedRunningTime="2026-11-02 14:09:20.219795001 +0000 UTC m=+725.403642337" Nov 02 14:09:21.067213 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:21.067174 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:21.185221 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:21.185187 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" event={"ID":"38537e17-d921-43b8-aa90-15a82b39342b","Type":"ContainerStarted","Data":"c27b3ae7e198db48b8d998d192d0dfb260ae118dcc30e1d46a9e8dc5c9a5416a"} Nov 02 14:09:21.836296 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:21.836251 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:09:21.837685 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:21.837655 4866 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 02 14:09:21.837836 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:21.837707 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" podUID="38537e17-d921-43b8-aa90-15a82b39342b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 02 14:09:22.836922 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:22.836881 4866 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 02 14:09:22.837462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:22.836964 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" podUID="38537e17-d921-43b8-aa90-15a82b39342b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 02 14:09:23.173971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:23.171202 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:09:23.216971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:23.212262 4866 generic.go:356] "Generic (PLEG): container finished" podID="b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" containerID="f57d6ee3c1381a21cf89a46a89b45153435a3f9c8285aeeaf8dfc37387b55e8b" exitCode=0 Nov 02 14:09:23.216971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:23.212452 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" event={"ID":"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910","Type":"ContainerDied","Data":"f57d6ee3c1381a21cf89a46a89b45153435a3f9c8285aeeaf8dfc37387b55e8b"} Nov 02 14:09:23.216971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:23.213029 4866 scope.go:122] "RemoveContainer" containerID="f57d6ee3c1381a21cf89a46a89b45153435a3f9c8285aeeaf8dfc37387b55e8b" Nov 02 14:09:23.216971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:23.214848 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:09:23.216971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:23.214966 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:09:23.216971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:23.215020 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:09:23.841970 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:23.839732 4866 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 02 14:09:23.841970 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:23.839852 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" podUID="38537e17-d921-43b8-aa90-15a82b39342b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 02 14:09:24.260972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:24.257618 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" event={"ID":"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910","Type":"ContainerStarted","Data":"3497d0eab8d1b5a104e6e248bf1e6118ed156cc7fd7f3ed8122f75557ef6ae1c"} Nov 02 14:09:24.841973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:24.841415 4866 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 02 14:09:24.841973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:24.841486 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" podUID="38537e17-d921-43b8-aa90-15a82b39342b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 02 14:09:24.912973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:24.908064 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:09:25.005102 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:25.003347 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 02 14:09:25.005102 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:25.003409 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" podUID="b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 02 14:09:25.023760 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:25.023725 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:09:25.354972 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:09:25.346642 4866 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861a94e0_defc_4d04_b11d_ecbdc66002b2.slice/crio-conmon-fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991.scope\": RecentStats: unable to find data in memory cache]" Nov 02 14:09:25.358975 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:09:25.358159 4866 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861a94e0_defc_4d04_b11d_ecbdc66002b2.slice/crio-conmon-fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991.scope\": RecentStats: unable to find data in memory cache]" Nov 02 14:09:25.842971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:25.841352 4866 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 02 14:09:25.842971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:25.841418 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" podUID="38537e17-d921-43b8-aa90-15a82b39342b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 02 14:09:25.916985 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:25.913351 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:09:25.988526 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:25.988493 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 02 14:09:25.988701 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:25.988547 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" podUID="b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 02 14:09:26.840877 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:26.840849 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:09:26.842011 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:26.841987 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:09:26.988770 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:26.988730 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 02 14:09:26.989273 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:26.988803 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" podUID="b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 02 14:09:27.048696 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:09:27.048653 4866 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861a94e0_defc_4d04_b11d_ecbdc66002b2.slice/crio-conmon-fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991.scope\": RecentStats: unable to find data in memory cache]" Nov 02 14:09:27.988401 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:27.988368 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 02 14:09:27.988575 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:27.988429 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" podUID="b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 02 14:09:28.988863 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:28.988826 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 02 14:09:28.989291 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:28.988892 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" podUID="b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 02 14:09:29.066970 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:29.066902 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:29.079770 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:29.079738 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:29.333638 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:29.333551 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:09:29.988664 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:29.988625 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 02 14:09:29.988844 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:29.988690 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" podUID="b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 02 14:09:30.993123 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:30.993089 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:09:30.994317 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:30.994289 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:09:33.036881 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:09:33.036842 4866 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861a94e0_defc_4d04_b11d_ecbdc66002b2.slice/crio-conmon-fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991.scope\": RecentStats: unable to find data in memory cache]" Nov 02 14:09:35.696314 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.696272 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs"] Nov 02 14:09:35.699892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.699870 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:35.719379 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.719353 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4be645\"/\"openshift-service-ca.crt\"" Nov 02 14:09:35.731421 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.731392 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4be645\"/\"kube-root-ca.crt\"" Nov 02 14:09:35.731847 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.731817 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4be645\"/\"simulator-w2-4be645-kserve-self-signed-certs\"" Nov 02 14:09:35.731937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.731918 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4be645\"/\"default-dockercfg-kwgz8\"" Nov 02 14:09:35.733907 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.733888 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs"] Nov 02 14:09:35.831466 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.831435 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-tmp-dir\") pod \"simulator-w2-4be645-kserve-6c7984cfc9-p4qxs\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:35.831649 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.831483 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0129df2e-9d21-45f7-9aa2-53777ce753b4-tls-certs\") pod \"simulator-w2-4be645-kserve-6c7984cfc9-p4qxs\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:35.831649 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.831614 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-model-cache\") pod \"simulator-w2-4be645-kserve-6c7984cfc9-p4qxs\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:35.831721 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.831662 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-home\") pod \"simulator-w2-4be645-kserve-6c7984cfc9-p4qxs\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:35.831758 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.831739 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g48w\" (UniqueName: \"kubernetes.io/projected/0129df2e-9d21-45f7-9aa2-53777ce753b4-kube-api-access-2g48w\") pod \"simulator-w2-4be645-kserve-6c7984cfc9-p4qxs\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:35.831791 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.831764 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-dshm\") pod \"simulator-w2-4be645-kserve-6c7984cfc9-p4qxs\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:35.933396 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.933356 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0129df2e-9d21-45f7-9aa2-53777ce753b4-tls-certs\") pod \"simulator-w2-4be645-kserve-6c7984cfc9-p4qxs\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:35.933566 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.933485 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-model-cache\") pod \"simulator-w2-4be645-kserve-6c7984cfc9-p4qxs\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:35.933566 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.933540 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-home\") pod \"simulator-w2-4be645-kserve-6c7984cfc9-p4qxs\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:35.933662 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.933597 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g48w\" (UniqueName: \"kubernetes.io/projected/0129df2e-9d21-45f7-9aa2-53777ce753b4-kube-api-access-2g48w\") pod \"simulator-w2-4be645-kserve-6c7984cfc9-p4qxs\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:35.933662 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.933625 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-dshm\") pod \"simulator-w2-4be645-kserve-6c7984cfc9-p4qxs\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:35.933777 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.933665 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-tmp-dir\") pod \"simulator-w2-4be645-kserve-6c7984cfc9-p4qxs\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:35.934062 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.934029 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-home\") pod \"simulator-w2-4be645-kserve-6c7984cfc9-p4qxs\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:35.934062 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.934058 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-tmp-dir\") pod \"simulator-w2-4be645-kserve-6c7984cfc9-p4qxs\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:35.934315 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.934293 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-model-cache\") pod \"simulator-w2-4be645-kserve-6c7984cfc9-p4qxs\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:35.936479 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.936448 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0129df2e-9d21-45f7-9aa2-53777ce753b4-tls-certs\") pod \"simulator-w2-4be645-kserve-6c7984cfc9-p4qxs\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:35.936589 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.936455 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-dshm\") pod \"simulator-w2-4be645-kserve-6c7984cfc9-p4qxs\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:35.960868 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:35.960788 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g48w\" (UniqueName: \"kubernetes.io/projected/0129df2e-9d21-45f7-9aa2-53777ce753b4-kube-api-access-2g48w\") pod \"simulator-w2-4be645-kserve-6c7984cfc9-p4qxs\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:36.009613 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:09:36.009579 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:09:36.011899 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:36.011882 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:36.308840 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:36.308817 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs"] Nov 02 14:09:36.311086 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:09:36.311054 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0129df2e_9d21_45f7_9aa2_53777ce753b4.slice/crio-837aa1304f6fefb518f6a73a72829791fa66c4967a72747060f5be379db5f231 WatchSource:0}: Error finding container 837aa1304f6fefb518f6a73a72829791fa66c4967a72747060f5be379db5f231: Status 404 returned error can't find the container with id 837aa1304f6fefb518f6a73a72829791fa66c4967a72747060f5be379db5f231 Nov 02 14:09:36.362133 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:36.362110 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" event={"ID":"0129df2e-9d21-45f7-9aa2-53777ce753b4","Type":"ContainerStarted","Data":"837aa1304f6fefb518f6a73a72829791fa66c4967a72747060f5be379db5f231"} Nov 02 14:09:37.061901 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:09:37.061869 4866 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861a94e0_defc_4d04_b11d_ecbdc66002b2.slice/crio-conmon-fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991.scope\": RecentStats: unable to find data in memory cache]" Nov 02 14:09:37.368553 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:37.368460 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" event={"ID":"0129df2e-9d21-45f7-9aa2-53777ce753b4","Type":"ContainerStarted","Data":"c63776dc143f1026221371d412fff10fbd92460fdc8db061db38b4a44f3f1f58"} Nov 02 14:09:37.417073 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:37.417020 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" podStartSLOduration=2.417001459 podStartE2EDuration="2.417001459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:37.415067641 +0000 UTC m=+742.598914977" watchObservedRunningTime="2026-11-02 14:09:37.417001459 +0000 UTC m=+742.600848797" Nov 02 14:09:38.013046 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:38.013011 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:46.012664 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:46.012618 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:46.024876 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:46.024853 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:46.450347 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:46.450324 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:09:47.123529 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:09:47.123485 4866 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861a94e0_defc_4d04_b11d_ecbdc66002b2.slice/crio-conmon-fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991.scope\": RecentStats: unable to find data in memory cache]" Nov 02 14:09:47.973247 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:09:47.973209 4866 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861a94e0_defc_4d04_b11d_ecbdc66002b2.slice/crio-conmon-fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991.scope\": RecentStats: unable to find data in memory cache]" Nov 02 14:09:53.559363 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.559331 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x"] Nov 02 14:09:53.562519 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.562497 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:53.590717 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.590686 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-555d1e\"/\"kube-root-ca.crt\"" Nov 02 14:09:53.590717 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.590701 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-555d1e\"/\"default-dockercfg-9dn26\"" Nov 02 14:09:53.590903 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.590748 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-555d1e\"/\"premium-w0-555d1e-kserve-self-signed-certs\"" Nov 02 14:09:53.598813 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.598784 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x"] Nov 02 14:09:53.679117 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.679088 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-555d1e\"/\"openshift-service-ca.crt\"" Nov 02 14:09:53.737763 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.737723 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-dshm\") pod \"premium-w0-555d1e-kserve-79486946df-n8m9x\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:53.737763 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.737771 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-model-cache\") pod \"premium-w0-555d1e-kserve-79486946df-n8m9x\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:53.738013 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.737887 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99j74\" (UniqueName: \"kubernetes.io/projected/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-kube-api-access-99j74\") pod \"premium-w0-555d1e-kserve-79486946df-n8m9x\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:53.738013 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.737984 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-tmp-dir\") pod \"premium-w0-555d1e-kserve-79486946df-n8m9x\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:53.738114 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.738036 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-tls-certs\") pod \"premium-w0-555d1e-kserve-79486946df-n8m9x\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:53.738175 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.738153 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-home\") pod \"premium-w0-555d1e-kserve-79486946df-n8m9x\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:53.839280 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.839200 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-dshm\") pod \"premium-w0-555d1e-kserve-79486946df-n8m9x\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:53.839280 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.839245 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-model-cache\") pod \"premium-w0-555d1e-kserve-79486946df-n8m9x\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:53.839489 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.839296 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99j74\" (UniqueName: \"kubernetes.io/projected/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-kube-api-access-99j74\") pod \"premium-w0-555d1e-kserve-79486946df-n8m9x\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:53.839489 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.839330 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-tmp-dir\") pod \"premium-w0-555d1e-kserve-79486946df-n8m9x\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:53.839489 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.839376 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-tls-certs\") pod \"premium-w0-555d1e-kserve-79486946df-n8m9x\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:53.839489 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.839438 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-home\") pod \"premium-w0-555d1e-kserve-79486946df-n8m9x\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:53.839778 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.839755 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-tmp-dir\") pod \"premium-w0-555d1e-kserve-79486946df-n8m9x\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:53.839873 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.839846 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-model-cache\") pod \"premium-w0-555d1e-kserve-79486946df-n8m9x\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:53.839961 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.839886 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-home\") pod \"premium-w0-555d1e-kserve-79486946df-n8m9x\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:53.841599 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.841577 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-dshm\") pod \"premium-w0-555d1e-kserve-79486946df-n8m9x\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:53.841866 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.841845 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-tls-certs\") pod \"premium-w0-555d1e-kserve-79486946df-n8m9x\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:53.867615 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.867586 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99j74\" (UniqueName: \"kubernetes.io/projected/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-kube-api-access-99j74\") pod \"premium-w0-555d1e-kserve-79486946df-n8m9x\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:53.875815 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:53.875584 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:54.181421 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:54.181394 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x"] Nov 02 14:09:54.183142 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:09:54.183116 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf7e11b5_e20c_4a71_b53b_1eb6b2de67fb.slice/crio-19ffdce1255d1cf0aa47df9ff059c6b29e68b919512872bde587ad58f0c59359 WatchSource:0}: Error finding container 19ffdce1255d1cf0aa47df9ff059c6b29e68b919512872bde587ad58f0c59359: Status 404 returned error can't find the container with id 19ffdce1255d1cf0aa47df9ff059c6b29e68b919512872bde587ad58f0c59359 Nov 02 14:09:54.481575 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:54.481529 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" event={"ID":"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb","Type":"ContainerStarted","Data":"81b79ec20f6323afff1689b8a8eeb98e68ca9c46dd7a5de411398d1864514bbc"} Nov 02 14:09:54.481575 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:54.481577 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" event={"ID":"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb","Type":"ContainerStarted","Data":"19ffdce1255d1cf0aa47df9ff059c6b29e68b919512872bde587ad58f0c59359"} Nov 02 14:09:54.524291 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:54.524240 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" podStartSLOduration=1.524227681 podStartE2EDuration="1.524227681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:54.522855003 +0000 UTC m=+759.706702342" watchObservedRunningTime="2026-11-02 14:09:54.524227681 +0000 UTC m=+759.708075018" Nov 02 14:09:54.876596 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:09:54.876515 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:09:57.182002 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:09:57.181963 4866 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861a94e0_defc_4d04_b11d_ecbdc66002b2.slice/crio-conmon-fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991.scope\": RecentStats: unable to find data in memory cache]" Nov 02 14:10:03.025421 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:10:03.025384 4866 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861a94e0_defc_4d04_b11d_ecbdc66002b2.slice/crio-conmon-fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991.scope\": RecentStats: unable to find data in memory cache]" Nov 02 14:10:03.876678 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:03.876638 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:10:03.889038 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:03.889007 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:10:04.546162 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:04.546131 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:10:06.010016 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:10:06.009976 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:10:07.194659 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:10:07.194624 4866 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861a94e0_defc_4d04_b11d_ecbdc66002b2.slice/crio-conmon-fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991.scope\": RecentStats: unable to find data in memory cache]" Nov 02 14:10:29.886928 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:29.886891 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts"] Nov 02 14:10:29.895061 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:29.895035 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" Nov 02 14:10:29.913459 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:29.913430 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts"] Nov 02 14:10:29.922448 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:29.922415 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-854b9e\"" Nov 02 14:10:29.922448 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:29.922415 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-854b9e-dockercfg-9v5dw\"" Nov 02 14:10:30.013387 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:30.013352 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgj8r\" (UniqueName: \"kubernetes.io/projected/8810f722-0ee3-41e5-a77a-ef9e5da7eeea-kube-api-access-dgj8r\") pod \"payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts\" (UID: \"8810f722-0ee3-41e5-a77a-ef9e5da7eeea\") " pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" Nov 02 14:10:30.013574 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:30.013418 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8810f722-0ee3-41e5-a77a-ef9e5da7eeea-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts\" (UID: \"8810f722-0ee3-41e5-a77a-ef9e5da7eeea\") " pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" Nov 02 14:10:30.114716 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:30.114678 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgj8r\" (UniqueName: \"kubernetes.io/projected/8810f722-0ee3-41e5-a77a-ef9e5da7eeea-kube-api-access-dgj8r\") pod \"payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts\" (UID: \"8810f722-0ee3-41e5-a77a-ef9e5da7eeea\") " pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" Nov 02 14:10:30.114932 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:30.114752 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8810f722-0ee3-41e5-a77a-ef9e5da7eeea-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts\" (UID: \"8810f722-0ee3-41e5-a77a-ef9e5da7eeea\") " pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" Nov 02 14:10:30.115503 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:30.115476 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8810f722-0ee3-41e5-a77a-ef9e5da7eeea-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts\" (UID: \"8810f722-0ee3-41e5-a77a-ef9e5da7eeea\") " pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" Nov 02 14:10:30.137934 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:30.137865 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgj8r\" (UniqueName: \"kubernetes.io/projected/8810f722-0ee3-41e5-a77a-ef9e5da7eeea-kube-api-access-dgj8r\") pod \"payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts\" (UID: \"8810f722-0ee3-41e5-a77a-ef9e5da7eeea\") " pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" Nov 02 14:10:30.209174 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:30.209144 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" Nov 02 14:10:30.507211 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:30.507181 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts"] Nov 02 14:10:30.509847 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:10:30.509809 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8810f722_0ee3_41e5_a77a_ef9e5da7eeea.slice/crio-e8472fd666991b43b0e11a27d77fb119db498f505cd2bff1f5782002c845f5c2 WatchSource:0}: Error finding container e8472fd666991b43b0e11a27d77fb119db498f505cd2bff1f5782002c845f5c2: Status 404 returned error can't find the container with id e8472fd666991b43b0e11a27d77fb119db498f505cd2bff1f5782002c845f5c2 Nov 02 14:10:30.686971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:30.684081 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" event={"ID":"8810f722-0ee3-41e5-a77a-ef9e5da7eeea","Type":"ContainerStarted","Data":"92afd22caae634e7c54bd19220a783c3a49c1b667b16589276093d2899faf60e"} Nov 02 14:10:30.686971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:30.684163 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" event={"ID":"8810f722-0ee3-41e5-a77a-ef9e5da7eeea","Type":"ContainerStarted","Data":"e8472fd666991b43b0e11a27d77fb119db498f505cd2bff1f5782002c845f5c2"} Nov 02 14:10:30.686971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:30.684195 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" Nov 02 14:10:30.728202 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:30.728145 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" podStartSLOduration=1.728130341 podStartE2EDuration="1.728130341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:10:30.724029769 +0000 UTC m=+795.907877105" watchObservedRunningTime="2026-11-02 14:10:30.728130341 +0000 UTC m=+795.911977693" Nov 02 14:10:36.009773 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:10:36.009737 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:10:39.325540 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.325508 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6"] Nov 02 14:10:39.328315 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.328291 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.354502 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.354477 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c0c28f\"/\"distinct-w4-c0c28f-kserve-self-signed-certs\"" Nov 02 14:10:39.356800 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.356770 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6"] Nov 02 14:10:39.510887 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.510846 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-home\") pod \"distinct-w4-c0c28f-kserve-dc4bc6557-md4q6\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.510887 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.510889 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-tmp-dir\") pod \"distinct-w4-c0c28f-kserve-dc4bc6557-md4q6\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.511137 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.510929 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7037eab-3f5f-419b-94f6-784ff7331925-tls-certs\") pod \"distinct-w4-c0c28f-kserve-dc4bc6557-md4q6\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.511137 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.511116 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq4zn\" (UniqueName: \"kubernetes.io/projected/d7037eab-3f5f-419b-94f6-784ff7331925-kube-api-access-cq4zn\") pod \"distinct-w4-c0c28f-kserve-dc4bc6557-md4q6\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.511223 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.511209 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-dshm\") pod \"distinct-w4-c0c28f-kserve-dc4bc6557-md4q6\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.511265 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.511249 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-model-cache\") pod \"distinct-w4-c0c28f-kserve-dc4bc6557-md4q6\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.612250 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.612161 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-dshm\") pod \"distinct-w4-c0c28f-kserve-dc4bc6557-md4q6\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.612250 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.612210 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-model-cache\") pod \"distinct-w4-c0c28f-kserve-dc4bc6557-md4q6\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.612250 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.612247 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-home\") pod \"distinct-w4-c0c28f-kserve-dc4bc6557-md4q6\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.612548 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.612264 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-tmp-dir\") pod \"distinct-w4-c0c28f-kserve-dc4bc6557-md4q6\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.612548 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.612290 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7037eab-3f5f-419b-94f6-784ff7331925-tls-certs\") pod \"distinct-w4-c0c28f-kserve-dc4bc6557-md4q6\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.612548 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.612340 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq4zn\" (UniqueName: \"kubernetes.io/projected/d7037eab-3f5f-419b-94f6-784ff7331925-kube-api-access-cq4zn\") pod \"distinct-w4-c0c28f-kserve-dc4bc6557-md4q6\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.612741 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.612718 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-home\") pod \"distinct-w4-c0c28f-kserve-dc4bc6557-md4q6\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.612801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.612764 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-model-cache\") pod \"distinct-w4-c0c28f-kserve-dc4bc6557-md4q6\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.612838 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.612820 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-tmp-dir\") pod \"distinct-w4-c0c28f-kserve-dc4bc6557-md4q6\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.614551 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.614522 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-dshm\") pod \"distinct-w4-c0c28f-kserve-dc4bc6557-md4q6\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.614871 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.614850 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7037eab-3f5f-419b-94f6-784ff7331925-tls-certs\") pod \"distinct-w4-c0c28f-kserve-dc4bc6557-md4q6\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.635917 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.635891 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq4zn\" (UniqueName: \"kubernetes.io/projected/d7037eab-3f5f-419b-94f6-784ff7331925-kube-api-access-cq4zn\") pod \"distinct-w4-c0c28f-kserve-dc4bc6557-md4q6\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.644798 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.644771 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:39.941357 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:39.941332 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6"] Nov 02 14:10:39.943759 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:10:39.943730 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7037eab_3f5f_419b_94f6_784ff7331925.slice/crio-abf571d0699e7db6acd193779ff250e0419659c3538c948c7d5955d9d6f310fc WatchSource:0}: Error finding container abf571d0699e7db6acd193779ff250e0419659c3538c948c7d5955d9d6f310fc: Status 404 returned error can't find the container with id abf571d0699e7db6acd193779ff250e0419659c3538c948c7d5955d9d6f310fc Nov 02 14:10:40.746106 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:40.746069 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" event={"ID":"d7037eab-3f5f-419b-94f6-784ff7331925","Type":"ContainerStarted","Data":"10be2c01de270a3c93942abe29b60816d4edbe45088b247941c692cd1cf1c579"} Nov 02 14:10:40.746505 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:40.746114 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" event={"ID":"d7037eab-3f5f-419b-94f6-784ff7331925","Type":"ContainerStarted","Data":"abf571d0699e7db6acd193779ff250e0419659c3538c948c7d5955d9d6f310fc"} Nov 02 14:10:40.790871 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:40.790810 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" podStartSLOduration=1.7907921340000001 podStartE2EDuration="1.790792134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:10:40.788403124 +0000 UTC m=+805.972250462" watchObservedRunningTime="2026-11-02 14:10:40.790792134 +0000 UTC m=+805.974639471" Nov 02 14:10:41.645728 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:41.645685 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:41.693014 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:41.692986 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" Nov 02 14:10:49.645412 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:49.645374 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:49.658697 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:49.658668 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:10:49.809840 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:10:49.809810 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:11:05.609847 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:05.609812 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7807-e92a4a"] Nov 02 14:11:05.613391 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:05.613370 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7807-e92a4a" Nov 02 14:11:05.631349 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:05.631311 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7807-e92a4a"] Nov 02 14:11:05.640583 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:05.640553 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nz2z6\"" Nov 02 14:11:05.709518 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:05.709478 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xj59\" (UniqueName: \"kubernetes.io/projected/7444eba2-6ca1-4c92-baf2-7da9a2475ec7-kube-api-access-5xj59\") pod \"test-curl-7807-e92a4a\" (UID: \"7444eba2-6ca1-4c92-baf2-7da9a2475ec7\") " pod="opendatahub/test-curl-7807-e92a4a" Nov 02 14:11:05.810211 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:05.810172 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xj59\" (UniqueName: \"kubernetes.io/projected/7444eba2-6ca1-4c92-baf2-7da9a2475ec7-kube-api-access-5xj59\") pod \"test-curl-7807-e92a4a\" (UID: \"7444eba2-6ca1-4c92-baf2-7da9a2475ec7\") " pod="opendatahub/test-curl-7807-e92a4a" Nov 02 14:11:05.833324 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:05.833298 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xj59\" (UniqueName: \"kubernetes.io/projected/7444eba2-6ca1-4c92-baf2-7da9a2475ec7-kube-api-access-5xj59\") pod \"test-curl-7807-e92a4a\" (UID: \"7444eba2-6ca1-4c92-baf2-7da9a2475ec7\") " pod="opendatahub/test-curl-7807-e92a4a" Nov 02 14:11:05.925526 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:05.925437 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7807-e92a4a" Nov 02 14:11:06.027500 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:11:06.027435 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:11:06.221029 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:06.221005 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7807-e92a4a"] Nov 02 14:11:06.222316 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:11:06.222294 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7444eba2_6ca1_4c92_baf2_7da9a2475ec7.slice/crio-35f49b0507856408af31bea1e29d94303847005486aac64300970188b23fe423 WatchSource:0}: Error finding container 35f49b0507856408af31bea1e29d94303847005486aac64300970188b23fe423: Status 404 returned error can't find the container with id 35f49b0507856408af31bea1e29d94303847005486aac64300970188b23fe423 Nov 02 14:11:06.902671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:06.902632 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7807-e92a4a" event={"ID":"7444eba2-6ca1-4c92-baf2-7da9a2475ec7","Type":"ContainerStarted","Data":"35f49b0507856408af31bea1e29d94303847005486aac64300970188b23fe423"} Nov 02 14:11:07.887025 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:07.886989 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx"] Nov 02 14:11:07.909780 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:07.909723 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:07.914697 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:07.914665 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx"] Nov 02 14:11:07.926484 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:07.926452 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-854b9e\"/\"test-model-854b9e-kserve-self-signed-certs\"" Nov 02 14:11:07.938531 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:07.938500 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-854b9e\"/\"openshift-service-ca.crt\"" Nov 02 14:11:07.938699 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:07.938605 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-854b9e\"/\"default-dockercfg-cxwwv\"" Nov 02 14:11:07.939336 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:07.939313 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-854b9e\"/\"kube-root-ca.crt\"" Nov 02 14:11:08.035039 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.035001 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-tmp-dir\") pod \"test-model-854b9e-kserve-7f77f88ccc-84qlx\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:08.035227 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.035062 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-tls-certs\") pod \"test-model-854b9e-kserve-7f77f88ccc-84qlx\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:08.035227 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.035093 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-home\") pod \"test-model-854b9e-kserve-7f77f88ccc-84qlx\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:08.035352 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.035241 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-model-cache\") pod \"test-model-854b9e-kserve-7f77f88ccc-84qlx\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:08.035352 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.035283 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-dshm\") pod \"test-model-854b9e-kserve-7f77f88ccc-84qlx\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:08.035465 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.035406 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9xrj\" (UniqueName: \"kubernetes.io/projected/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-kube-api-access-d9xrj\") pod \"test-model-854b9e-kserve-7f77f88ccc-84qlx\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:08.137150 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.136514 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-model-cache\") pod \"test-model-854b9e-kserve-7f77f88ccc-84qlx\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:08.137150 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.136573 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-dshm\") pod \"test-model-854b9e-kserve-7f77f88ccc-84qlx\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:08.137150 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.137046 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-model-cache\") pod \"test-model-854b9e-kserve-7f77f88ccc-84qlx\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:08.138476 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.137551 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9xrj\" (UniqueName: \"kubernetes.io/projected/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-kube-api-access-d9xrj\") pod \"test-model-854b9e-kserve-7f77f88ccc-84qlx\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:08.138476 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.137692 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-tmp-dir\") pod \"test-model-854b9e-kserve-7f77f88ccc-84qlx\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:08.138476 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.137722 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-tls-certs\") pod \"test-model-854b9e-kserve-7f77f88ccc-84qlx\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:08.138476 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.137748 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-home\") pod \"test-model-854b9e-kserve-7f77f88ccc-84qlx\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:08.138476 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.138147 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-home\") pod \"test-model-854b9e-kserve-7f77f88ccc-84qlx\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:08.139081 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.139034 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-tmp-dir\") pod \"test-model-854b9e-kserve-7f77f88ccc-84qlx\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:08.149262 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.149113 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-dshm\") pod \"test-model-854b9e-kserve-7f77f88ccc-84qlx\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:08.149422 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.149397 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-tls-certs\") pod \"test-model-854b9e-kserve-7f77f88ccc-84qlx\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:08.163050 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.163021 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9xrj\" (UniqueName: \"kubernetes.io/projected/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-kube-api-access-d9xrj\") pod \"test-model-854b9e-kserve-7f77f88ccc-84qlx\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:08.227134 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.227099 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:08.550732 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.550702 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx"] Nov 02 14:11:08.552450 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:11:08.552413 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b7821c4_51bb_4d23_ae57_c2f4803cbacb.slice/crio-7c0721489eac1ceb1cc89e606d18cbaa9e1e35f12e7b1a7a6d7b7854cb97131d WatchSource:0}: Error finding container 7c0721489eac1ceb1cc89e606d18cbaa9e1e35f12e7b1a7a6d7b7854cb97131d: Status 404 returned error can't find the container with id 7c0721489eac1ceb1cc89e606d18cbaa9e1e35f12e7b1a7a6d7b7854cb97131d Nov 02 14:11:08.920462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.920366 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" event={"ID":"8b7821c4-51bb-4d23-ae57-c2f4803cbacb","Type":"ContainerStarted","Data":"22ab9adda97879bee8155a9a80a754d8aba800d50da860d8737172cbecd98838"} Nov 02 14:11:08.920462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.920419 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" event={"ID":"8b7821c4-51bb-4d23-ae57-c2f4803cbacb","Type":"ContainerStarted","Data":"7c0721489eac1ceb1cc89e606d18cbaa9e1e35f12e7b1a7a6d7b7854cb97131d"} Nov 02 14:11:08.921876 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.921853 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7807-e92a4a" event={"ID":"7444eba2-6ca1-4c92-baf2-7da9a2475ec7","Type":"ContainerStarted","Data":"790328d27e311ddd9a3f5b9fb6d9ac0308fb338c13e2a082519aafbe903138f1"} Nov 02 14:11:08.967217 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:08.967169 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" podStartSLOduration=1.967155418 podStartE2EDuration="1.967155418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:08.964640495 +0000 UTC m=+834.148487832" watchObservedRunningTime="2026-11-02 14:11:08.967155418 +0000 UTC m=+834.151002754" Nov 02 14:11:09.006768 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:09.006711 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7807-e92a4a" podStartSLOduration=1.621687061 podStartE2EDuration="4.006696707s" totalImagesPullingTime="2.385009646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:09.006410778 +0000 UTC m=+834.190258115" watchObservedRunningTime="2026-11-02 14:11:09.006696707 +0000 UTC m=+834.190544045" Nov 02 14:11:09.227584 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:09.227551 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:09.899543 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:09.899510 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7807-e92a4a"] Nov 02 14:11:09.904010 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:09.903973 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7807-e92a4a"] Nov 02 14:11:10.072349 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:10.072315 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7807-2c2ad3"] Nov 02 14:11:10.072783 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:10.072770 4866 cpu_manager.go:414] "containerMap: removing container" podUID="7444eba2-6ca1-4c92-baf2-7da9a2475ec7" containerName="test-curl-7807-e92a4a" Nov 02 14:11:10.072832 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:10.072784 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7444eba2-6ca1-4c92-baf2-7da9a2475ec7" containerName="test-curl-7807-e92a4a" Nov 02 14:11:10.072899 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:10.072889 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="7444eba2-6ca1-4c92-baf2-7da9a2475ec7" containerName="test-curl-7807-e92a4a" Nov 02 14:11:10.076284 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:10.076262 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7807-2c2ad3" Nov 02 14:11:10.096249 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:10.096214 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7807-2c2ad3"] Nov 02 14:11:10.162550 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:10.162464 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l69lv\" (UniqueName: \"kubernetes.io/projected/e892476a-033b-4ec6-bb6e-be2415f9a906-kube-api-access-l69lv\") pod \"test-curl-7807-2c2ad3\" (UID: \"e892476a-033b-4ec6-bb6e-be2415f9a906\") " pod="opendatahub/test-curl-7807-2c2ad3" Nov 02 14:11:10.264013 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:10.263975 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l69lv\" (UniqueName: \"kubernetes.io/projected/e892476a-033b-4ec6-bb6e-be2415f9a906-kube-api-access-l69lv\") pod \"test-curl-7807-2c2ad3\" (UID: \"e892476a-033b-4ec6-bb6e-be2415f9a906\") " pod="opendatahub/test-curl-7807-2c2ad3" Nov 02 14:11:10.288167 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:10.288133 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l69lv\" (UniqueName: \"kubernetes.io/projected/e892476a-033b-4ec6-bb6e-be2415f9a906-kube-api-access-l69lv\") pod \"test-curl-7807-2c2ad3\" (UID: \"e892476a-033b-4ec6-bb6e-be2415f9a906\") " pod="opendatahub/test-curl-7807-2c2ad3" Nov 02 14:11:10.388269 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:10.388224 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7807-2c2ad3" Nov 02 14:11:10.688673 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:10.688643 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7807-2c2ad3"] Nov 02 14:11:10.690521 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:11:10.690491 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode892476a_033b_4ec6_bb6e_be2415f9a906.slice/crio-55451ca2b8b8bbb8e3049c3f0a8ff50df6bc667fb101e5626fd2a1a90d3a4be7 WatchSource:0}: Error finding container 55451ca2b8b8bbb8e3049c3f0a8ff50df6bc667fb101e5626fd2a1a90d3a4be7: Status 404 returned error can't find the container with id 55451ca2b8b8bbb8e3049c3f0a8ff50df6bc667fb101e5626fd2a1a90d3a4be7 Nov 02 14:11:10.936404 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:10.936368 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7807-2c2ad3" event={"ID":"e892476a-033b-4ec6-bb6e-be2415f9a906","Type":"ContainerStarted","Data":"557844a58b761d55e26e038cc3f3a3d3e3ebc575310e8fabd0b1e09bea887de5"} Nov 02 14:11:10.936404 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:10.936415 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7807-2c2ad3" event={"ID":"e892476a-033b-4ec6-bb6e-be2415f9a906","Type":"ContainerStarted","Data":"55451ca2b8b8bbb8e3049c3f0a8ff50df6bc667fb101e5626fd2a1a90d3a4be7"} Nov 02 14:11:10.936773 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:10.936744 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7807-e92a4a" podUID="7444eba2-6ca1-4c92-baf2-7da9a2475ec7" containerName="test-curl-7807-e92a4a" containerID="cri-o://790328d27e311ddd9a3f5b9fb6d9ac0308fb338c13e2a082519aafbe903138f1" gracePeriod=2 Nov 02 14:11:10.976079 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:10.976025 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7807-2c2ad3" podStartSLOduration=0.976010206 podStartE2EDuration="976.010206ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:10.974718272 +0000 UTC m=+836.158565629" watchObservedRunningTime="2026-11-02 14:11:10.976010206 +0000 UTC m=+836.159857542" Nov 02 14:11:12.173132 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:12.173093 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7807-2c2ad3"] Nov 02 14:11:12.176641 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:12.176608 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7807-2c2ad3"] Nov 02 14:11:12.947430 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:12.947391 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7807-2c2ad3" podUID="e892476a-033b-4ec6-bb6e-be2415f9a906" containerName="test-curl-7807-2c2ad3" containerID="cri-o://557844a58b761d55e26e038cc3f3a3d3e3ebc575310e8fabd0b1e09bea887de5" gracePeriod=2 Nov 02 14:11:13.107924 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:13.107893 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7807-e92a4a" Nov 02 14:11:13.305646 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:13.305566 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7444eba2-6ca1-4c92-baf2-7da9a2475ec7-kube-api-access-5xj59\" (UniqueName: \"kubernetes.io/projected/7444eba2-6ca1-4c92-baf2-7da9a2475ec7-kube-api-access-5xj59\") pod \"7444eba2-6ca1-4c92-baf2-7da9a2475ec7\" (UID: \"7444eba2-6ca1-4c92-baf2-7da9a2475ec7\") " Nov 02 14:11:13.307626 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:13.307590 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7444eba2-6ca1-4c92-baf2-7da9a2475ec7-kube-api-access-5xj59" pod "7444eba2-6ca1-4c92-baf2-7da9a2475ec7" (UID: "7444eba2-6ca1-4c92-baf2-7da9a2475ec7"). InnerVolumeSpecName "kube-api-access-5xj59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:13.407068 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:13.407034 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xj59\" (UniqueName: \"kubernetes.io/projected/7444eba2-6ca1-4c92-baf2-7da9a2475ec7-kube-api-access-5xj59\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:11:13.953864 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:13.953829 4866 generic.go:356] "Generic (PLEG): container finished" podID="7444eba2-6ca1-4c92-baf2-7da9a2475ec7" containerID="790328d27e311ddd9a3f5b9fb6d9ac0308fb338c13e2a082519aafbe903138f1" exitCode=137 Nov 02 14:11:13.954047 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:13.953876 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7807-e92a4a" Nov 02 14:11:13.954047 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:13.953902 4866 scope.go:122] "RemoveContainer" containerID="790328d27e311ddd9a3f5b9fb6d9ac0308fb338c13e2a082519aafbe903138f1" Nov 02 14:11:13.962821 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:13.962796 4866 scope.go:122] "RemoveContainer" containerID="790328d27e311ddd9a3f5b9fb6d9ac0308fb338c13e2a082519aafbe903138f1" Nov 02 14:11:13.963115 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:11:13.963095 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"790328d27e311ddd9a3f5b9fb6d9ac0308fb338c13e2a082519aafbe903138f1\": container with ID starting with 790328d27e311ddd9a3f5b9fb6d9ac0308fb338c13e2a082519aafbe903138f1 not found: ID does not exist" containerID="790328d27e311ddd9a3f5b9fb6d9ac0308fb338c13e2a082519aafbe903138f1" Nov 02 14:11:13.963189 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:13.963123 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790328d27e311ddd9a3f5b9fb6d9ac0308fb338c13e2a082519aafbe903138f1"} err="failed to get container status \"790328d27e311ddd9a3f5b9fb6d9ac0308fb338c13e2a082519aafbe903138f1\": rpc error: code = NotFound desc = could not find container \"790328d27e311ddd9a3f5b9fb6d9ac0308fb338c13e2a082519aafbe903138f1\": container with ID starting with 790328d27e311ddd9a3f5b9fb6d9ac0308fb338c13e2a082519aafbe903138f1 not found: ID does not exist" Nov 02 14:11:13.976161 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:11:13.976129 4866 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7807-e92a4a\" is forbidden: User \"system:node:ip-10-0-134-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-99.ec2.internal' and this object" podUID="7444eba2-6ca1-4c92-baf2-7da9a2475ec7" pod="opendatahub/test-curl-7807-e92a4a" Nov 02 14:11:14.354802 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.354719 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd"] Nov 02 14:11:14.355456 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.355434 4866 cpu_manager.go:414] "containerMap: removing container" podUID="e892476a-033b-4ec6-bb6e-be2415f9a906" containerName="test-curl-7807-2c2ad3" Nov 02 14:11:14.355556 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.355457 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e892476a-033b-4ec6-bb6e-be2415f9a906" containerName="test-curl-7807-2c2ad3" Nov 02 14:11:14.355609 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.355588 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="e892476a-033b-4ec6-bb6e-be2415f9a906" containerName="test-curl-7807-2c2ad3" Nov 02 14:11:14.360214 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.360192 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.376728 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:11:14.376692 4866 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7807-e92a4a\" is forbidden: User \"system:node:ip-10-0-134-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-99.ec2.internal' and this object" podUID="7444eba2-6ca1-4c92-baf2-7da9a2475ec7" pod="opendatahub/test-curl-7807-e92a4a" Nov 02 14:11:14.380939 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.380918 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-bf0b0d-openshift-default-dockercfg-bjslb\"" Nov 02 14:11:14.389789 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.389764 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd"] Nov 02 14:11:14.518308 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.518272 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-credential-socket\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.518497 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.518320 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-token\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.518497 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.518431 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-podinfo\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.518611 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.518523 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6xvn\" (UniqueName: \"kubernetes.io/projected/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-kube-api-access-w6xvn\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.518611 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.518600 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-workload-socket\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.518694 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.518625 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-data\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.518735 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.518717 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istiod-ca-cert\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.518767 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.518754 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-workload-certs\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.518934 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.518901 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-envoy\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.619848 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.619762 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-podinfo\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.619848 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.619819 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6xvn\" (UniqueName: \"kubernetes.io/projected/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-kube-api-access-w6xvn\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.619848 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.619845 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-workload-socket\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.620182 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.619861 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-data\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.620182 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.619889 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istiod-ca-cert\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.620182 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.619913 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-workload-certs\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.620182 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.619987 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-envoy\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.620182 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.620045 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-credential-socket\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.620182 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.620079 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-token\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.620469 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.620420 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-workload-socket\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.620530 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.620501 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-workload-certs\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.620589 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.620539 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-data\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.620685 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.620666 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-credential-socket\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.620834 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.620814 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istiod-ca-cert\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.622153 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.622134 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-envoy\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.622412 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.622395 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-podinfo\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.644265 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.644237 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-token\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.644789 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.644766 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6xvn\" (UniqueName: \"kubernetes.io/projected/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-kube-api-access-w6xvn\") pod \"e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.673820 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.673788 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:14.945887 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:14.945856 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7444eba2-6ca1-4c92-baf2-7da9a2475ec7" path="/var/lib/kubelet/pods/7444eba2-6ca1-4c92-baf2-7da9a2475ec7/volumes" Nov 02 14:11:14.954600 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:11:14.954573 4866 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7807-e92a4a\" is forbidden: User \"system:node:ip-10-0-134-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-99.ec2.internal' and this object" podUID="7444eba2-6ca1-4c92-baf2-7da9a2475ec7" pod="opendatahub/test-curl-7807-e92a4a" Nov 02 14:11:15.002726 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:15.002695 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd"] Nov 02 14:11:15.007608 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:11:15.007575 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a6adc3_f95e_42e0_99cc_344f7cfa647c.slice/crio-801f667a60320858e82d29c68fe21b4e4c89daef024329823f4f280a76c20630 WatchSource:0}: Error finding container 801f667a60320858e82d29c68fe21b4e4c89daef024329823f4f280a76c20630: Status 404 returned error can't find the container with id 801f667a60320858e82d29c68fe21b4e4c89daef024329823f4f280a76c20630 Nov 02 14:11:15.010596 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:15.010561 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:11:15.010693 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:15.010637 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:11:15.010693 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:15.010679 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:11:15.154795 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:15.154761 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7807-2c2ad3" Nov 02 14:11:15.225829 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:15.225748 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e892476a-033b-4ec6-bb6e-be2415f9a906-kube-api-access-l69lv\" (UniqueName: \"kubernetes.io/projected/e892476a-033b-4ec6-bb6e-be2415f9a906-kube-api-access-l69lv\") pod \"e892476a-033b-4ec6-bb6e-be2415f9a906\" (UID: \"e892476a-033b-4ec6-bb6e-be2415f9a906\") " Nov 02 14:11:15.227795 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:15.227766 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e892476a-033b-4ec6-bb6e-be2415f9a906-kube-api-access-l69lv" pod "e892476a-033b-4ec6-bb6e-be2415f9a906" (UID: "e892476a-033b-4ec6-bb6e-be2415f9a906"). InnerVolumeSpecName "kube-api-access-l69lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:15.326919 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:15.326885 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l69lv\" (UniqueName: \"kubernetes.io/projected/e892476a-033b-4ec6-bb6e-be2415f9a906-kube-api-access-l69lv\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:11:15.966223 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:15.966186 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" event={"ID":"e2a6adc3-f95e-42e0-99cc-344f7cfa647c","Type":"ContainerStarted","Data":"097e45e3e4827aba353e5137c0e539551967372656fc4fdd01a774da664111e7"} Nov 02 14:11:15.966705 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:15.966291 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" event={"ID":"e2a6adc3-f95e-42e0-99cc-344f7cfa647c","Type":"ContainerStarted","Data":"801f667a60320858e82d29c68fe21b4e4c89daef024329823f4f280a76c20630"} Nov 02 14:11:15.967535 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:15.967508 4866 generic.go:356] "Generic (PLEG): container finished" podID="e892476a-033b-4ec6-bb6e-be2415f9a906" containerID="557844a58b761d55e26e038cc3f3a3d3e3ebc575310e8fabd0b1e09bea887de5" exitCode=137 Nov 02 14:11:15.967665 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:15.967537 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7807-2c2ad3" Nov 02 14:11:15.967665 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:15.967568 4866 scope.go:122] "RemoveContainer" containerID="557844a58b761d55e26e038cc3f3a3d3e3ebc575310e8fabd0b1e09bea887de5" Nov 02 14:11:15.977236 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:15.977219 4866 scope.go:122] "RemoveContainer" containerID="557844a58b761d55e26e038cc3f3a3d3e3ebc575310e8fabd0b1e09bea887de5" Nov 02 14:11:15.977558 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:11:15.977538 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"557844a58b761d55e26e038cc3f3a3d3e3ebc575310e8fabd0b1e09bea887de5\": container with ID starting with 557844a58b761d55e26e038cc3f3a3d3e3ebc575310e8fabd0b1e09bea887de5 not found: ID does not exist" containerID="557844a58b761d55e26e038cc3f3a3d3e3ebc575310e8fabd0b1e09bea887de5" Nov 02 14:11:15.977623 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:15.977566 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"557844a58b761d55e26e038cc3f3a3d3e3ebc575310e8fabd0b1e09bea887de5"} err="failed to get container status \"557844a58b761d55e26e038cc3f3a3d3e3ebc575310e8fabd0b1e09bea887de5\": rpc error: code = NotFound desc = could not find container \"557844a58b761d55e26e038cc3f3a3d3e3ebc575310e8fabd0b1e09bea887de5\": container with ID starting with 557844a58b761d55e26e038cc3f3a3d3e3ebc575310e8fabd0b1e09bea887de5 not found: ID does not exist" Nov 02 14:11:16.025622 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:16.025560 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" podStartSLOduration=2.02554305 podStartE2EDuration="2.02554305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:16.023659767 +0000 UTC m=+841.207507107" watchObservedRunningTime="2026-11-02 14:11:16.02554305 +0000 UTC m=+841.209390388" Nov 02 14:11:16.037439 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:11:16.037402 4866 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7807-2c2ad3\" is forbidden: User \"system:node:ip-10-0-134-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-99.ec2.internal' and this object" podUID="e892476a-033b-4ec6-bb6e-be2415f9a906" pod="opendatahub/test-curl-7807-2c2ad3" Nov 02 14:11:16.674897 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:16.674857 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:16.680677 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:16.680650 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:16.750084 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:11:16.750042 4866 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7807-2c2ad3\" is forbidden: User \"system:node:ip-10-0-134-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-99.ec2.internal' and this object" podUID="e892476a-033b-4ec6-bb6e-be2415f9a906" pod="opendatahub/test-curl-7807-2c2ad3" Nov 02 14:11:16.945579 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:16.945543 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e892476a-033b-4ec6-bb6e-be2415f9a906" path="/var/lib/kubelet/pods/e892476a-033b-4ec6-bb6e-be2415f9a906/volumes" Nov 02 14:11:16.973975 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:16.973933 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:16.975015 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:16.974992 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:18.228328 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:18.228283 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:18.241006 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:18.240977 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:18.996072 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:18.996037 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:11:36.012215 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:11:36.012128 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:11:37.735396 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.735359 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk"] Nov 02 14:11:37.739491 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.739469 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.760561 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.760527 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c0c28f-openshift-default-dockercfg-z9fnh\"" Nov 02 14:11:37.786736 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.786698 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk"] Nov 02 14:11:37.886768 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.886723 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.886998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.886781 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.886998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.886883 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.886998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.886924 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.886998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.886977 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.887232 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.887030 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.887232 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.887094 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4659ae-3de7-4902-9fdf-98373b40172c-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.887232 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.887152 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-276gh\" (UniqueName: \"kubernetes.io/projected/4e4659ae-3de7-4902-9fdf-98373b40172c-kube-api-access-276gh\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.887232 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.887184 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.988046 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.987936 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.988202 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.988088 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.988202 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.988141 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.988202 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.988191 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.988378 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.988317 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.988378 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.988338 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.988481 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.988391 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.988481 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.988431 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.988481 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.988465 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4659ae-3de7-4902-9fdf-98373b40172c-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.988638 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.988510 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-276gh\" (UniqueName: \"kubernetes.io/projected/4e4659ae-3de7-4902-9fdf-98373b40172c-kube-api-access-276gh\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.988638 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.988556 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.988638 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.988579 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.988791 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.988645 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.989214 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.989189 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4659ae-3de7-4902-9fdf-98373b40172c-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.990517 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.990491 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:37.990979 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:37.990937 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:38.022064 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:38.022027 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:38.022224 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:38.022112 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-276gh\" (UniqueName: \"kubernetes.io/projected/4e4659ae-3de7-4902-9fdf-98373b40172c-kube-api-access-276gh\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:38.052804 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:38.052766 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:38.384724 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:38.384693 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk"] Nov 02 14:11:38.385979 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:11:38.385933 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4659ae_3de7_4902_9fdf_98373b40172c.slice/crio-8751718059de7449307ce25d44a289eb72115852a6130eb276708d9861ca349e WatchSource:0}: Error finding container 8751718059de7449307ce25d44a289eb72115852a6130eb276708d9861ca349e: Status 404 returned error can't find the container with id 8751718059de7449307ce25d44a289eb72115852a6130eb276708d9861ca349e Nov 02 14:11:38.388380 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:38.388349 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:11:38.388449 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:38.388413 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:11:38.388449 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:38.388442 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:11:39.153780 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:39.153741 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" event={"ID":"4e4659ae-3de7-4902-9fdf-98373b40172c","Type":"ContainerStarted","Data":"63d0196c9c9e8594c6a3e4c934fc9da87fab1f46286cb7ce053de3c4e74cdd87"} Nov 02 14:11:39.154202 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:39.153798 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" event={"ID":"4e4659ae-3de7-4902-9fdf-98373b40172c","Type":"ContainerStarted","Data":"8751718059de7449307ce25d44a289eb72115852a6130eb276708d9861ca349e"} Nov 02 14:11:40.053441 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:40.053401 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:40.055496 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:40.055466 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 02 14:11:40.055620 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:40.055524 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" podUID="4e4659ae-3de7-4902-9fdf-98373b40172c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 02 14:11:41.053546 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:41.053500 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 02 14:11:41.054119 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:41.053568 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" podUID="4e4659ae-3de7-4902-9fdf-98373b40172c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 02 14:11:42.053872 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:42.053829 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 02 14:11:42.054312 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:42.053900 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" podUID="4e4659ae-3de7-4902-9fdf-98373b40172c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 02 14:11:43.053640 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:43.053599 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 02 14:11:43.053836 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:43.053666 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" podUID="4e4659ae-3de7-4902-9fdf-98373b40172c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 02 14:11:44.053549 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:44.053503 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 02 14:11:44.053985 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:44.053566 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" podUID="4e4659ae-3de7-4902-9fdf-98373b40172c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 02 14:11:45.059393 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:45.059350 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 02 14:11:45.066496 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:45.065122 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" podUID="4e4659ae-3de7-4902-9fdf-98373b40172c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 02 14:11:46.058176 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:46.058144 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:46.058561 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:46.058542 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:46.059458 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:46.059438 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:11:46.131673 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:46.131614 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" podStartSLOduration=9.131593528 podStartE2EDuration="9.131593528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:39.223412812 +0000 UTC m=+864.407260154" watchObservedRunningTime="2026-11-02 14:11:46.131593528 +0000 UTC m=+871.315440866" Nov 02 14:11:50.833462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.833422 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq"] Nov 02 14:11:50.841425 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.841399 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.862684 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.862654 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-2bb4ef-openshift-default-dockercfg-rlxt2\"" Nov 02 14:11:50.877373 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.877338 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq"] Nov 02 14:11:50.894728 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.894690 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-workload-certs\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.894904 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.894758 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-token\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.894904 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.894890 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-credential-socket\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.895017 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.894917 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-envoy\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.895017 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.894966 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istiod-ca-cert\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.895122 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.895100 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-podinfo\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.895257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.895239 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-data\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.895325 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.895310 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmzvh\" (UniqueName: \"kubernetes.io/projected/60bf0c3a-284a-49d1-af2c-34ea34cf44af-kube-api-access-fmzvh\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.895362 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.895341 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-workload-socket\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.996106 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.996060 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-data\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.996323 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.996156 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmzvh\" (UniqueName: \"kubernetes.io/projected/60bf0c3a-284a-49d1-af2c-34ea34cf44af-kube-api-access-fmzvh\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.996323 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.996188 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-workload-socket\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.996323 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.996231 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-workload-certs\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.996323 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.996274 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-token\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.996323 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.996310 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-credential-socket\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.996576 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.996335 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-envoy\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.996576 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.996364 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istiod-ca-cert\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.996576 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.996446 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-podinfo\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.996576 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.996524 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-data\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.996748 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.996725 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-credential-socket\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.997044 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.997016 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-workload-socket\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.997293 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.997270 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-workload-certs\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:50.997643 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:50.997622 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istiod-ca-cert\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:51.032270 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:51.032238 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-envoy\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:51.033280 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:51.033249 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-token\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:51.033417 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:51.033255 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-podinfo\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:51.034484 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:51.034461 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmzvh\" (UniqueName: \"kubernetes.io/projected/60bf0c3a-284a-49d1-af2c-34ea34cf44af-kube-api-access-fmzvh\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:51.156373 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:51.156294 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:51.496197 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:51.496171 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq"] Nov 02 14:11:51.497886 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:11:51.497861 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60bf0c3a_284a_49d1_af2c_34ea34cf44af.slice/crio-ece69a75386bdc9bc86ea02c6c937814bcd864dbc778b5db531f87ebebb71856 WatchSource:0}: Error finding container ece69a75386bdc9bc86ea02c6c937814bcd864dbc778b5db531f87ebebb71856: Status 404 returned error can't find the container with id ece69a75386bdc9bc86ea02c6c937814bcd864dbc778b5db531f87ebebb71856 Nov 02 14:11:51.524157 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:51.524120 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:11:51.524264 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:51.524189 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:11:51.524264 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:51.524220 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:11:52.249639 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:52.249540 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" event={"ID":"60bf0c3a-284a-49d1-af2c-34ea34cf44af","Type":"ContainerStarted","Data":"4a3880a31c24cd484f8de21f5780d9248715609323f29cbce2fc2d121fb74652"} Nov 02 14:11:52.249639 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:52.249607 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" event={"ID":"60bf0c3a-284a-49d1-af2c-34ea34cf44af","Type":"ContainerStarted","Data":"ece69a75386bdc9bc86ea02c6c937814bcd864dbc778b5db531f87ebebb71856"} Nov 02 14:11:52.312061 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:52.311997 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" podStartSLOduration=2.311976625 podStartE2EDuration="2.311976625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:52.308736273 +0000 UTC m=+877.492583611" watchObservedRunningTime="2026-11-02 14:11:52.311976625 +0000 UTC m=+877.495823963" Nov 02 14:11:52.949304 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:52.949266 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg"] Nov 02 14:11:52.953752 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:52.953729 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:52.980654 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:52.980604 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg"] Nov 02 14:11:53.055850 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.055806 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.055850 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.055854 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.056113 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.055882 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.056113 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.056000 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.056225 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.056147 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.056334 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.056269 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s29r\" (UniqueName: \"kubernetes.io/projected/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-kube-api-access-2s29r\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.056334 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.056312 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.056487 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.056440 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.056487 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.056472 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.156618 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.156575 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:53.157096 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.157068 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.157164 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.157123 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.157164 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.157141 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.157265 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.157167 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.157265 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.157185 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.157353 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.157283 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.157353 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.157342 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s29r\" (UniqueName: \"kubernetes.io/projected/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-kube-api-access-2s29r\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.157442 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.157380 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.157523 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.157478 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.157575 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.157547 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.157641 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.157619 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.157927 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.157902 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.158086 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.158059 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.158478 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.158452 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.160203 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.160176 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.160409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.160391 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.163504 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.163484 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:53.181616 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.181587 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.182079 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.182060 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s29r\" (UniqueName: \"kubernetes.io/projected/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-kube-api-access-2s29r\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.256688 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.256612 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:53.257836 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.257810 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:11:53.269684 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.269657 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:53.603041 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.603018 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg"] Nov 02 14:11:53.605018 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:11:53.604989 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac034d4_7e4f_468d_8a4d_b712f4d81bae.slice/crio-15869d1c629c106ee6468e842badb7178121cba4c315967855a655c8494494f4 WatchSource:0}: Error finding container 15869d1c629c106ee6468e842badb7178121cba4c315967855a655c8494494f4: Status 404 returned error can't find the container with id 15869d1c629c106ee6468e842badb7178121cba4c315967855a655c8494494f4 Nov 02 14:11:53.607694 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.607658 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:11:53.607793 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.607773 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:11:53.607844 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:53.607829 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:11:54.263677 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:54.263641 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" event={"ID":"8ac034d4-7e4f-468d-8a4d-b712f4d81bae","Type":"ContainerStarted","Data":"fe245455cbf9c36166526f4b712cb01d1d7bc73d40e473db06c2f9322aae81fa"} Nov 02 14:11:54.264197 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:54.263699 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" event={"ID":"8ac034d4-7e4f-468d-8a4d-b712f4d81bae","Type":"ContainerStarted","Data":"15869d1c629c106ee6468e842badb7178121cba4c315967855a655c8494494f4"} Nov 02 14:11:54.270182 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:54.270157 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:11:54.271671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:54.271650 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 02 14:11:54.271758 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:54.271698 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" podUID="8ac034d4-7e4f-468d-8a4d-b712f4d81bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 02 14:11:54.333278 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:54.333083 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" podStartSLOduration=2.333063385 podStartE2EDuration="2.333063385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:54.325405906 +0000 UTC m=+879.509253243" watchObservedRunningTime="2026-11-02 14:11:54.333063385 +0000 UTC m=+879.516910722" Nov 02 14:11:55.270089 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:55.270059 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 02 14:11:55.270522 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:55.270109 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" podUID="8ac034d4-7e4f-468d-8a4d-b712f4d81bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 02 14:11:56.270163 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:56.270122 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 02 14:11:56.270663 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:56.270188 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" podUID="8ac034d4-7e4f-468d-8a4d-b712f4d81bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 02 14:11:56.276094 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:56.276066 4866 generic.go:356] "Generic (PLEG): container finished" podID="e2a6adc3-f95e-42e0-99cc-344f7cfa647c" containerID="097e45e3e4827aba353e5137c0e539551967372656fc4fdd01a774da664111e7" exitCode=0 Nov 02 14:11:56.276240 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:56.276137 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" event={"ID":"e2a6adc3-f95e-42e0-99cc-344f7cfa647c","Type":"ContainerDied","Data":"097e45e3e4827aba353e5137c0e539551967372656fc4fdd01a774da664111e7"} Nov 02 14:11:56.276639 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:56.276624 4866 scope.go:122] "RemoveContainer" containerID="097e45e3e4827aba353e5137c0e539551967372656fc4fdd01a774da664111e7" Nov 02 14:11:56.277869 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:56.277845 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:11:56.277922 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:56.277908 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:11:56.277983 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:56.277939 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:11:57.270807 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:57.270768 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 02 14:11:57.271309 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:57.270818 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" podUID="8ac034d4-7e4f-468d-8a4d-b712f4d81bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 02 14:11:57.283646 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:57.283613 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" event={"ID":"e2a6adc3-f95e-42e0-99cc-344f7cfa647c","Type":"ContainerStarted","Data":"a43d2bec590a42c353db53144d928eb0fdfd40526a06acd906fc9d950871d7b5"} Nov 02 14:11:57.674025 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:57.673930 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:11:57.675273 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:57.675251 4866 patch_prober.go:29] interesting pod/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 14:11:57.675430 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:57.675413 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" podUID="e2a6adc3-f95e-42e0-99cc-344f7cfa647c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 14:11:58.270537 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:58.270507 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 02 14:11:58.273989 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:58.272216 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" podUID="8ac034d4-7e4f-468d-8a4d-b712f4d81bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 02 14:11:58.678026 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:58.677372 4866 patch_prober.go:29] interesting pod/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 14:11:58.678026 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:58.677471 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" podUID="e2a6adc3-f95e-42e0-99cc-344f7cfa647c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 14:11:59.270392 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:59.270354 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 02 14:11:59.270699 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:59.270669 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" podUID="8ac034d4-7e4f-468d-8a4d-b712f4d81bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 02 14:11:59.689090 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:59.688376 4866 patch_prober.go:29] interesting pod/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 14:11:59.689090 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:59.688453 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" podUID="e2a6adc3-f95e-42e0-99cc-344f7cfa647c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 14:11:59.883617 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:11:59.883542 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" podUID="d7037eab-3f5f-419b-94f6-784ff7331925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:12:00.271008 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:00.270931 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 02 14:12:00.271225 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:00.271058 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" podUID="8ac034d4-7e4f-468d-8a4d-b712f4d81bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 02 14:12:00.674930 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:00.674828 4866 patch_prober.go:29] interesting pod/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 14:12:00.675103 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:00.674917 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" podUID="e2a6adc3-f95e-42e0-99cc-344f7cfa647c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 14:12:01.658375 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:01.658343 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:12:01.659076 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:01.659055 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:12:01.659807 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:01.659785 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:12:01.674329 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:01.674291 4866 patch_prober.go:29] interesting pod/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 14:12:01.674526 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:01.674355 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" podUID="e2a6adc3-f95e-42e0-99cc-344f7cfa647c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 14:12:01.974175 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:01.974141 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:12:02.674554 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:02.674513 4866 patch_prober.go:29] interesting pod/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 14:12:02.675016 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:02.674582 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" podUID="e2a6adc3-f95e-42e0-99cc-344f7cfa647c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 14:12:03.679104 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:03.679070 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:12:03.680138 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:03.680116 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:12:06.010758 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:12:06.010723 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:12:15.011712 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:15.011676 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:12:15.014137 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:15.014110 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:12:22.881356 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:22.881309 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2"] Nov 02 14:12:22.884999 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:22.884975 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:22.937775 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:22.937742 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2"] Nov 02 14:12:22.978793 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:22.978754 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:22.979004 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:22.978811 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:22.979053 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:22.979014 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb4ld\" (UniqueName: \"kubernetes.io/projected/87976167-09be-407f-aa64-6e6a384739ce-kube-api-access-gb4ld\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:22.979089 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:22.979068 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:22.979136 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:22.979121 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:22.979171 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:22.979154 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:22.979371 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:22.979356 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87976167-09be-407f-aa64-6e6a384739ce-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:22.979436 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:22.979416 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87976167-09be-407f-aa64-6e6a384739ce-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:22.979473 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:22.979457 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87976167-09be-407f-aa64-6e6a384739ce-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.081078 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.081040 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.081248 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.081105 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.081248 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.081190 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb4ld\" (UniqueName: \"kubernetes.io/projected/87976167-09be-407f-aa64-6e6a384739ce-kube-api-access-gb4ld\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.081248 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.081229 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.081392 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.081266 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.081392 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.081294 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.081392 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.081355 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87976167-09be-407f-aa64-6e6a384739ce-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.081540 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.081392 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87976167-09be-407f-aa64-6e6a384739ce-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.081540 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.081435 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87976167-09be-407f-aa64-6e6a384739ce-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.081648 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.081584 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.081704 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.081649 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.081807 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.081772 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.081969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.081932 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.082372 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.082346 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87976167-09be-407f-aa64-6e6a384739ce-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.083627 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.083605 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.083837 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.083819 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87976167-09be-407f-aa64-6e6a384739ce-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.123871 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.123831 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb4ld\" (UniqueName: \"kubernetes.io/projected/87976167-09be-407f-aa64-6e6a384739ce-kube-api-access-gb4ld\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.126000 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.125978 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87976167-09be-407f-aa64-6e6a384739ce-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.237342 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.237302 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:23.571238 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.571072 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2"] Nov 02 14:12:23.574657 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:12:23.574622 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87976167_09be_407f_aa64_6e6a384739ce.slice/crio-8e3e5fc818d5fe238771250da6b4d8ac15ce6b982371c2f35a5c2d26c55f3fa0 WatchSource:0}: Error finding container 8e3e5fc818d5fe238771250da6b4d8ac15ce6b982371c2f35a5c2d26c55f3fa0: Status 404 returned error can't find the container with id 8e3e5fc818d5fe238771250da6b4d8ac15ce6b982371c2f35a5c2d26c55f3fa0 Nov 02 14:12:23.577172 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.577139 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:12:23.577283 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.577232 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:12:23.577350 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:23.577279 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:12:24.539587 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:24.539549 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" event={"ID":"87976167-09be-407f-aa64-6e6a384739ce","Type":"ContainerStarted","Data":"d981cf9e4f1ed98a35dfa628fef62b3c132aea3eb1b36c020df75bca55c65c19"} Nov 02 14:12:24.540094 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:24.539644 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" event={"ID":"87976167-09be-407f-aa64-6e6a384739ce","Type":"ContainerStarted","Data":"8e3e5fc818d5fe238771250da6b4d8ac15ce6b982371c2f35a5c2d26c55f3fa0"} Nov 02 14:12:24.627002 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:24.614320 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" podStartSLOduration=2.6142993900000002 podStartE2EDuration="2.61429939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:24.607373847 +0000 UTC m=+909.791221186" watchObservedRunningTime="2026-11-02 14:12:24.61429939 +0000 UTC m=+909.798146728" Nov 02 14:12:25.237473 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:25.237439 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:25.239307 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:25.239281 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 14:12:25.239452 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:25.239337 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" podUID="87976167-09be-407f-aa64-6e6a384739ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 14:12:26.252994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:26.249091 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 14:12:26.252994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:26.249176 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" podUID="87976167-09be-407f-aa64-6e6a384739ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 14:12:27.238493 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:27.238445 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 14:12:27.238678 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:27.238515 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" podUID="87976167-09be-407f-aa64-6e6a384739ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 14:12:28.237640 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:28.237604 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 14:12:28.238167 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:28.237678 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" podUID="87976167-09be-407f-aa64-6e6a384739ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 14:12:29.238291 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:29.238247 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 14:12:29.238854 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:29.238331 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" podUID="87976167-09be-407f-aa64-6e6a384739ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 14:12:29.573121 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:29.573040 4866 generic.go:356] "Generic (PLEG): container finished" podID="60bf0c3a-284a-49d1-af2c-34ea34cf44af" containerID="4a3880a31c24cd484f8de21f5780d9248715609323f29cbce2fc2d121fb74652" exitCode=0 Nov 02 14:12:29.573275 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:29.573120 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" event={"ID":"60bf0c3a-284a-49d1-af2c-34ea34cf44af","Type":"ContainerDied","Data":"4a3880a31c24cd484f8de21f5780d9248715609323f29cbce2fc2d121fb74652"} Nov 02 14:12:29.573626 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:29.573609 4866 scope.go:122] "RemoveContainer" containerID="4a3880a31c24cd484f8de21f5780d9248715609323f29cbce2fc2d121fb74652" Nov 02 14:12:29.575052 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:29.575025 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:12:29.575161 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:29.575112 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:12:29.575234 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:29.575162 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:12:30.241431 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:30.241394 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 14:12:30.242071 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:30.242030 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" podUID="87976167-09be-407f-aa64-6e6a384739ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 14:12:30.581282 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:30.581181 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" event={"ID":"60bf0c3a-284a-49d1-af2c-34ea34cf44af","Type":"ContainerStarted","Data":"fe1c8c8795250e0668ff08f44d14227feada0b1896849fa575f41514531233cb"} Nov 02 14:12:31.156983 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:31.156932 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:12:31.157245 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:31.157222 4866 patch_prober.go:29] interesting pod/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 02 14:12:31.157312 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:31.157288 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" podUID="60bf0c3a-284a-49d1-af2c-34ea34cf44af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 02 14:12:31.242107 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:31.242075 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:31.242524 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:31.242373 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:31.243177 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:31.243158 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:12:32.157308 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:32.157275 4866 patch_prober.go:29] interesting pod/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 02 14:12:32.157458 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:32.157338 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" podUID="60bf0c3a-284a-49d1-af2c-34ea34cf44af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 02 14:12:33.156971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:33.156576 4866 patch_prober.go:29] interesting pod/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 02 14:12:33.156971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:33.156664 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" podUID="60bf0c3a-284a-49d1-af2c-34ea34cf44af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 02 14:12:34.180972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:34.180543 4866 patch_prober.go:29] interesting pod/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 02 14:12:34.180972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:34.180622 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" podUID="60bf0c3a-284a-49d1-af2c-34ea34cf44af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 02 14:12:35.157123 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.157083 4866 patch_prober.go:29] interesting pod/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 02 14:12:35.157323 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.157162 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" podUID="60bf0c3a-284a-49d1-af2c-34ea34cf44af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 02 14:12:35.417031 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.398729 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59"] Nov 02 14:12:35.426992 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.422572 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" Nov 02 14:12:35.428780 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.428751 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59"] Nov 02 14:12:35.430428 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.430401 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59\" (UID: \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" Nov 02 14:12:35.430544 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.430454 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg7xc\" (UniqueName: \"kubernetes.io/projected/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-kube-api-access-hg7xc\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59\" (UID: \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" Nov 02 14:12:35.430598 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.430542 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59\" (UID: \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" Nov 02 14:12:35.430641 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.430611 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59\" (UID: \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" Nov 02 14:12:35.437823 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.437793 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"agc-cred-f650d0393004\"" Nov 02 14:12:35.438073 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.438053 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"payload-processing-external-model-e2e--c0104787-dockercfg-tx5xz\"" Nov 02 14:12:35.449606 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.449576 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"routing-overlay\"" Nov 02 14:12:35.449898 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.449605 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2bb4ef\"" Nov 02 14:12:35.450091 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.449727 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"kube-root-ca.crt\"" Nov 02 14:12:35.450091 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.449891 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"openshift-service-ca.crt\"" Nov 02 14:12:35.531858 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.531824 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59\" (UID: \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" Nov 02 14:12:35.532106 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.531876 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg7xc\" (UniqueName: \"kubernetes.io/projected/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-kube-api-access-hg7xc\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59\" (UID: \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" Nov 02 14:12:35.532106 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.531916 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59\" (UID: \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" Nov 02 14:12:35.532106 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.531988 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59\" (UID: \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" Nov 02 14:12:35.532742 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.532714 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59\" (UID: \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" Nov 02 14:12:35.532865 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.532807 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59\" (UID: \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" Nov 02 14:12:35.534530 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.534508 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59\" (UID: \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" Nov 02 14:12:35.594078 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.593691 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg7xc\" (UniqueName: \"kubernetes.io/projected/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-kube-api-access-hg7xc\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59\" (UID: \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" Nov 02 14:12:35.742416 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:35.742374 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" Nov 02 14:12:36.027320 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:12:36.027223 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:12:36.068413 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:36.068388 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59"] Nov 02 14:12:36.071375 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:12:36.071342 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50eee8c0_0f94_4633_9732_bfdd1d6c9f68.slice/crio-c94135adfbd16ff4c3b324856186090f644623d4f969a7712c574a9e667bd582 WatchSource:0}: Error finding container c94135adfbd16ff4c3b324856186090f644623d4f969a7712c574a9e667bd582: Status 404 returned error can't find the container with id c94135adfbd16ff4c3b324856186090f644623d4f969a7712c574a9e667bd582 Nov 02 14:12:36.156784 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:36.156745 4866 patch_prober.go:29] interesting pod/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 02 14:12:36.156984 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:36.156815 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" podUID="60bf0c3a-284a-49d1-af2c-34ea34cf44af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 02 14:12:36.685984 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:36.685924 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" event={"ID":"50eee8c0-0f94-4633-9732-bfdd1d6c9f68","Type":"ContainerStarted","Data":"b5994338f13637bfb380b38dbe6e928c4b9b618e37f2bc4b1f88264fd5988a34"} Nov 02 14:12:36.686377 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:36.686110 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" Nov 02 14:12:36.686377 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:36.686150 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" event={"ID":"50eee8c0-0f94-4633-9732-bfdd1d6c9f68","Type":"ContainerStarted","Data":"c94135adfbd16ff4c3b324856186090f644623d4f969a7712c574a9e667bd582"} Nov 02 14:12:36.731638 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:36.731584 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" podStartSLOduration=1.731569975 podStartE2EDuration="1.731569975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:36.730104978 +0000 UTC m=+921.913952315" watchObservedRunningTime="2026-11-02 14:12:36.731569975 +0000 UTC m=+921.915417312" Nov 02 14:12:37.161290 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.161256 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:12:37.161801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.161774 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:12:37.162656 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.162635 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:12:37.913247 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.913213 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m"] Nov 02 14:12:37.917282 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.917261 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:37.927342 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.927320 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s"] Nov 02 14:12:37.930831 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.930815 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:37.942770 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.942743 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m"] Nov 02 14:12:37.956238 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.956207 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s"] Nov 02 14:12:37.963932 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.963901 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:37.964102 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.963962 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:37.964102 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.964080 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:37.964221 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.964150 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:37.964310 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.964271 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:37.964364 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.964327 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:37.964437 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.964418 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:37.964493 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.964478 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:37.964613 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.964587 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:37.964654 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.964638 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:37.964695 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.964674 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:37.964741 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.964726 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g82fz\" (UniqueName: \"kubernetes.io/projected/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-kube-api-access-g82fz\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:37.964797 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.964769 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:37.965674 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.965643 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1faca5-3a0e-4a93-afda-78b9aa799050-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:37.965802 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.965691 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:37.965802 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.965720 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:37.965802 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.965790 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-797cr\" (UniqueName: \"kubernetes.io/projected/9a1faca5-3a0e-4a93-afda-78b9aa799050-kube-api-access-797cr\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:37.965992 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:37.965820 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.066937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.066903 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.067148 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.066989 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:38.067148 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.067044 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.067148 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.067080 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:38.067328 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.067289 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.067391 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.067352 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:38.067462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.067440 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:38.067510 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.067449 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.067557 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.067512 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82fz\" (UniqueName: \"kubernetes.io/projected/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-kube-api-access-g82fz\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.067680 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.067656 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.067793 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.067775 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1faca5-3a0e-4a93-afda-78b9aa799050-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:38.067878 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.067805 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.067878 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.067840 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.067878 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.067887 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-797cr\" (UniqueName: \"kubernetes.io/projected/9a1faca5-3a0e-4a93-afda-78b9aa799050-kube-api-access-797cr\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:38.068085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.067898 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.068085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.067912 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.068085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.067989 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:38.068085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.068019 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:38.068085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.068059 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.068322 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.067779 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:38.068322 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.068128 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:38.068422 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.068399 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.068482 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.067805 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.068537 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.068475 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:38.068537 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.067841 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:38.068537 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.068477 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1faca5-3a0e-4a93-afda-78b9aa799050-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:38.068784 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.068766 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:38.068882 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.068859 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.070635 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.070613 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:38.070722 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.070644 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.070722 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.070685 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:38.070809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.070787 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.088941 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.088910 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84"] Nov 02 14:12:38.094019 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.093998 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.094125 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.094013 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82fz\" (UniqueName: \"kubernetes.io/projected/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-kube-api-access-g82fz\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.100141 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.100112 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:38.100257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.100112 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.100257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.100221 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-797cr\" (UniqueName: \"kubernetes.io/projected/9a1faca5-3a0e-4a93-afda-78b9aa799050-kube-api-access-797cr\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:38.122498 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.122463 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84"] Nov 02 14:12:38.169892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.169797 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.169892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.169845 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rdgl\" (UniqueName: \"kubernetes.io/projected/ebd8b16b-08fa-4933-8451-5e9660510c50-kube-api-access-5rdgl\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.170102 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.169993 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.170102 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.170096 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.170210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.170164 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd8b16b-08fa-4933-8451-5e9660510c50-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.170274 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.170213 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.170359 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.170344 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.170421 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.170397 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.170475 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.170462 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.232918 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.232886 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:38.245830 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.243568 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:38.271918 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.271882 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.272121 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.271966 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd8b16b-08fa-4933-8451-5e9660510c50-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.272121 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.272005 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.272121 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.272086 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.272282 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.272135 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.272282 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.272183 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.272282 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.272230 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.272282 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.272255 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rdgl\" (UniqueName: \"kubernetes.io/projected/ebd8b16b-08fa-4933-8451-5e9660510c50-kube-api-access-5rdgl\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.272514 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.272320 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.272833 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.272733 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.272833 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.272818 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd8b16b-08fa-4933-8451-5e9660510c50-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.273084 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.273026 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.273295 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.273273 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.273412 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.273367 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.275120 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.275100 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.276712 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.276693 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.304995 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.304792 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rdgl\" (UniqueName: \"kubernetes.io/projected/ebd8b16b-08fa-4933-8451-5e9660510c50-kube-api-access-5rdgl\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.307586 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.307558 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.437036 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.436927 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:38.580190 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.580158 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m"] Nov 02 14:12:38.581232 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:12:38.581209 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ae3055_5b22_4b7e_bee4_d853a0b8517b.slice/crio-3076c42167014cd71d21c6bc679f786ed6ab7ce4601f7686a05a1348f2eab0f8 WatchSource:0}: Error finding container 3076c42167014cd71d21c6bc679f786ed6ab7ce4601f7686a05a1348f2eab0f8: Status 404 returned error can't find the container with id 3076c42167014cd71d21c6bc679f786ed6ab7ce4601f7686a05a1348f2eab0f8 Nov 02 14:12:38.581868 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:12:38.581855 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1faca5_3a0e_4a93_afda_78b9aa799050.slice/crio-e974320e3b2407a1242ef2eda1e4e0bdc778e44f7bf0dcf03a4e5db191a8777d WatchSource:0}: Error finding container e974320e3b2407a1242ef2eda1e4e0bdc778e44f7bf0dcf03a4e5db191a8777d: Status 404 returned error can't find the container with id e974320e3b2407a1242ef2eda1e4e0bdc778e44f7bf0dcf03a4e5db191a8777d Nov 02 14:12:38.582935 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.582909 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s"] Nov 02 14:12:38.583934 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.583899 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:12:38.584092 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.584014 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:12:38.584092 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.584063 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:12:38.584460 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.584432 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:12:38.584689 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.584495 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:12:38.584689 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.584542 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:12:38.704863 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.704585 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" event={"ID":"9a1faca5-3a0e-4a93-afda-78b9aa799050","Type":"ContainerStarted","Data":"84ef9de0c7303f2610b22e64dc5d5c169d4839772c09628d0e4a45fabf089fdd"} Nov 02 14:12:38.704863 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.704647 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" event={"ID":"9a1faca5-3a0e-4a93-afda-78b9aa799050","Type":"ContainerStarted","Data":"e974320e3b2407a1242ef2eda1e4e0bdc778e44f7bf0dcf03a4e5db191a8777d"} Nov 02 14:12:38.707090 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.707003 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" event={"ID":"a9ae3055-5b22-4b7e-bee4-d853a0b8517b","Type":"ContainerStarted","Data":"092da1b100bd2d58520e27e9a026b8078fdb875e0fa0badbe16e61371a60b1f8"} Nov 02 14:12:38.707090 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.707057 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" event={"ID":"a9ae3055-5b22-4b7e-bee4-d853a0b8517b","Type":"ContainerStarted","Data":"3076c42167014cd71d21c6bc679f786ed6ab7ce4601f7686a05a1348f2eab0f8"} Nov 02 14:12:38.750883 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.750846 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84"] Nov 02 14:12:38.755433 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.755168 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" podStartSLOduration=1.755150354 podStartE2EDuration="1.755150354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:38.752841591 +0000 UTC m=+923.936688927" watchObservedRunningTime="2026-11-02 14:12:38.755150354 +0000 UTC m=+923.938997692" Nov 02 14:12:38.756214 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:12:38.756192 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd8b16b_08fa_4933_8451_5e9660510c50.slice/crio-d8957374d0c97326244def51f6293f26016f1d8c5c7fc0f5d8d32249c4a167be WatchSource:0}: Error finding container d8957374d0c97326244def51f6293f26016f1d8c5c7fc0f5d8d32249c4a167be: Status 404 returned error can't find the container with id d8957374d0c97326244def51f6293f26016f1d8c5c7fc0f5d8d32249c4a167be Nov 02 14:12:38.759826 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.759468 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:12:38.759826 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.759556 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:12:38.759826 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.759599 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:12:38.800748 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:38.800694 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" podStartSLOduration=1.8006799199999999 podStartE2EDuration="1.80067992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:38.799267211 +0000 UTC m=+923.983114549" watchObservedRunningTime="2026-11-02 14:12:38.80067992 +0000 UTC m=+923.984527257" Nov 02 14:12:39.233872 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:39.233839 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:39.235560 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:39.235536 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 02 14:12:39.235674 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:39.235581 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" podUID="a9ae3055-5b22-4b7e-bee4-d853a0b8517b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 02 14:12:39.243837 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:39.243811 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:39.245095 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:39.245074 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 14:12:39.245159 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:39.245117 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" podUID="9a1faca5-3a0e-4a93-afda-78b9aa799050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 14:12:39.747403 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:39.747363 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" event={"ID":"ebd8b16b-08fa-4933-8451-5e9660510c50","Type":"ContainerStarted","Data":"bffe2d99b14e6d2fea40598c4f4dd1e0330347abc722723c615c28e75b020cd6"} Nov 02 14:12:39.748535 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:39.748509 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" event={"ID":"ebd8b16b-08fa-4933-8451-5e9660510c50","Type":"ContainerStarted","Data":"d8957374d0c97326244def51f6293f26016f1d8c5c7fc0f5d8d32249c4a167be"} Nov 02 14:12:39.797289 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:39.797234 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" podStartSLOduration=1.797215224 podStartE2EDuration="1.797215224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:39.793255657 +0000 UTC m=+924.977102994" watchObservedRunningTime="2026-11-02 14:12:39.797215224 +0000 UTC m=+924.981062567" Nov 02 14:12:40.235970 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:40.235418 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 02 14:12:40.235970 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:40.235485 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" podUID="a9ae3055-5b22-4b7e-bee4-d853a0b8517b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 02 14:12:40.244969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:40.244416 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 14:12:40.244969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:40.244482 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" podUID="9a1faca5-3a0e-4a93-afda-78b9aa799050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 14:12:40.458167 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:40.458116 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 14:12:40.458438 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:40.458407 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" podUID="ebd8b16b-08fa-4933-8451-5e9660510c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 14:12:40.458844 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:40.458822 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:41.236423 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:41.236378 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 02 14:12:41.237097 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:41.237061 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" podUID="a9ae3055-5b22-4b7e-bee4-d853a0b8517b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 02 14:12:41.266777 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:41.266741 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 14:12:41.267081 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:41.267057 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" podUID="9a1faca5-3a0e-4a93-afda-78b9aa799050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 14:12:41.441134 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:41.441099 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 14:12:41.441410 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:41.441381 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" podUID="ebd8b16b-08fa-4933-8451-5e9660510c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 14:12:42.233434 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:42.233393 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 02 14:12:42.233716 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:42.233683 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" podUID="a9ae3055-5b22-4b7e-bee4-d853a0b8517b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 02 14:12:42.257814 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:42.257779 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 14:12:42.258387 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:42.258355 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" podUID="9a1faca5-3a0e-4a93-afda-78b9aa799050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 14:12:42.446380 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:42.446338 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 14:12:42.446677 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:42.446645 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" podUID="ebd8b16b-08fa-4933-8451-5e9660510c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 14:12:42.544033 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:42.543400 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" podUID="df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:12:42.571100 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:42.571049 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" podUID="6d89a28a-7f87-4924-93c3-48ae7d11e56e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:12:42.811597 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:42.811502 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" podUID="d7037eab-3f5f-419b-94f6-784ff7331925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:12:42.996506 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:42.994590 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" podUID="8b7821c4-51bb-4d23-ae57-c2f4803cbacb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:12:43.233452 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:43.233403 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 02 14:12:43.233720 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:43.233686 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" podUID="a9ae3055-5b22-4b7e-bee4-d853a0b8517b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 02 14:12:43.246366 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:43.246328 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 14:12:43.246609 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:43.246575 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" podUID="9a1faca5-3a0e-4a93-afda-78b9aa799050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 14:12:43.313579 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:43.313536 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" podUID="9753ddaa-3899-46ea-952f-5f03d76037c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:12:43.438647 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:43.438594 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 14:12:43.438936 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:43.438890 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" podUID="ebd8b16b-08fa-4933-8451-5e9660510c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 14:12:43.544190 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:43.544059 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" podUID="df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 02 14:12:43.547156 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:43.547124 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:12:43.575875 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:43.573037 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" podUID="6d89a28a-7f87-4924-93c3-48ae7d11e56e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:12:43.575875 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:43.573784 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:12:43.893282 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:43.893199 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:12:43.896322 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:43.893257 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" podUID="d7037eab-3f5f-419b-94f6-784ff7331925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:12:43.900129 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:43.900107 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:12:44.003435 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:44.003392 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" podUID="8b7821c4-51bb-4d23-ae57-c2f4803cbacb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:12:44.004427 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:44.004404 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:12:44.293756 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:44.293716 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 02 14:12:44.294170 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:44.294136 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" podUID="a9ae3055-5b22-4b7e-bee4-d853a0b8517b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 02 14:12:44.294411 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:44.294391 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 14:12:44.294569 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:44.294547 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" podUID="9a1faca5-3a0e-4a93-afda-78b9aa799050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 14:12:44.381972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:44.378639 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" podUID="9753ddaa-3899-46ea-952f-5f03d76037c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 02 14:12:44.381972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:44.379042 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:12:44.478432 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:44.478312 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" podUID="0129df2e-9d21-45f7-9aa2-53777ce753b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Nov 02 14:12:44.493577 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:44.493457 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 14:12:44.493577 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:44.493512 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" podUID="ebd8b16b-08fa-4933-8451-5e9660510c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 14:12:44.584733 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:44.584566 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:12:44.598022 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:44.594170 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:12:44.658097 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:44.658050 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" podUID="df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:12:44.796499 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:44.796465 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:12:44.797758 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:44.797568 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:12:45.259972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:45.259718 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 14:12:45.259972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:45.259779 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" podUID="9a1faca5-3a0e-4a93-afda-78b9aa799050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 14:12:45.281033 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:45.260505 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 02 14:12:45.281033 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:45.260547 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" podUID="a9ae3055-5b22-4b7e-bee4-d853a0b8517b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 02 14:12:45.441973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:45.438428 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 14:12:45.441973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:45.438495 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" podUID="ebd8b16b-08fa-4933-8451-5e9660510c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 14:12:46.234042 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:46.234001 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 02 14:12:46.234267 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:46.234069 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" podUID="a9ae3055-5b22-4b7e-bee4-d853a0b8517b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 02 14:12:46.244648 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:46.244609 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 14:12:46.244940 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:46.244904 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" podUID="9a1faca5-3a0e-4a93-afda-78b9aa799050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 14:12:46.448613 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:46.448582 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:46.449714 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:46.449693 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:46.451382 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:46.451356 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:12:47.696412 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:47.696385 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" Nov 02 14:12:48.281169 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:48.281122 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:12:48.281330 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:48.281205 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" podUID="a9ae3055-5b22-4b7e-bee4-d853a0b8517b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:12:48.286058 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:48.286031 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:12:48.286193 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:48.286072 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" podUID="9a1faca5-3a0e-4a93-afda-78b9aa799050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:12:48.447358 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:48.447321 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:48.449081 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:48.448882 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:48.454144 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:48.453969 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:12:48.482001 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:48.480093 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:48.482001 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:48.481133 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:12:48.482968 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:12:48.482929 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:13:06.021443 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:13:06.021350 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:13:14.482288 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:14.482251 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh"] Nov 02 14:13:14.487692 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:14.487667 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" Nov 02 14:13:14.508043 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:14.508005 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh"] Nov 02 14:13:14.530061 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:14.530029 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-de8cc7-serving-cert\"" Nov 02 14:13:14.581263 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:14.581229 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqhmd\" (UniqueName: \"kubernetes.io/projected/27f22147-746a-4848-bfe4-b31f89025f07-kube-api-access-dqhmd\") pod \"maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh\" (UID: \"27f22147-746a-4848-bfe4-b31f89025f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" Nov 02 14:13:14.581444 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:14.581368 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f22147-746a-4848-bfe4-b31f89025f07-metrics-certs\") pod \"maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh\" (UID: \"27f22147-746a-4848-bfe4-b31f89025f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" Nov 02 14:13:14.581444 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:14.581407 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27f22147-746a-4848-bfe4-b31f89025f07-maas-api-tls\") pod \"maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh\" (UID: \"27f22147-746a-4848-bfe4-b31f89025f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" Nov 02 14:13:14.581562 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:14.581547 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f22147-746a-4848-bfe4-b31f89025f07-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh\" (UID: \"27f22147-746a-4848-bfe4-b31f89025f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" Nov 02 14:13:14.682603 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:14.682563 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f22147-746a-4848-bfe4-b31f89025f07-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh\" (UID: \"27f22147-746a-4848-bfe4-b31f89025f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" Nov 02 14:13:14.682788 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:14.682633 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqhmd\" (UniqueName: \"kubernetes.io/projected/27f22147-746a-4848-bfe4-b31f89025f07-kube-api-access-dqhmd\") pod \"maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh\" (UID: \"27f22147-746a-4848-bfe4-b31f89025f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" Nov 02 14:13:14.682788 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:14.682738 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f22147-746a-4848-bfe4-b31f89025f07-metrics-certs\") pod \"maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh\" (UID: \"27f22147-746a-4848-bfe4-b31f89025f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" Nov 02 14:13:14.682788 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:14.682776 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27f22147-746a-4848-bfe4-b31f89025f07-maas-api-tls\") pod \"maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh\" (UID: \"27f22147-746a-4848-bfe4-b31f89025f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" Nov 02 14:13:14.682975 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:13:14.682942 4866 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-serving-cert: secret "maas-api-e2e-praxis-a-de8cc7-serving-cert" not found Nov 02 14:13:14.683043 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:13:14.683031 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f22147-746a-4848-bfe4-b31f89025f07-maas-api-tls podName:27f22147-746a-4848-bfe4-b31f89025f07 nodeName:}" failed. No retries permitted until 2026-11-02 14:13:15.183009337 +0000 UTC m=+960.366856664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/27f22147-746a-4848-bfe4-b31f89025f07-maas-api-tls") pod "maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" (UID: "27f22147-746a-4848-bfe4-b31f89025f07") : secret "maas-api-e2e-praxis-a-de8cc7-serving-cert" not found Nov 02 14:13:14.683475 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:14.683448 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f22147-746a-4848-bfe4-b31f89025f07-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh\" (UID: \"27f22147-746a-4848-bfe4-b31f89025f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" Nov 02 14:13:14.685799 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:14.685763 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f22147-746a-4848-bfe4-b31f89025f07-metrics-certs\") pod \"maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh\" (UID: \"27f22147-746a-4848-bfe4-b31f89025f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" Nov 02 14:13:14.737654 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:14.737534 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqhmd\" (UniqueName: \"kubernetes.io/projected/27f22147-746a-4848-bfe4-b31f89025f07-kube-api-access-dqhmd\") pod \"maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh\" (UID: \"27f22147-746a-4848-bfe4-b31f89025f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" Nov 02 14:13:15.206976 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:15.196159 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27f22147-746a-4848-bfe4-b31f89025f07-maas-api-tls\") pod \"maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh\" (UID: \"27f22147-746a-4848-bfe4-b31f89025f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" Nov 02 14:13:15.233966 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:15.216405 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27f22147-746a-4848-bfe4-b31f89025f07-maas-api-tls\") pod \"maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh\" (UID: \"27f22147-746a-4848-bfe4-b31f89025f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" Nov 02 14:13:15.404690 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:15.404651 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" Nov 02 14:13:15.713637 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:15.713543 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh"] Nov 02 14:13:15.717854 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:13:15.717830 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f22147_746a_4848_bfe4_b31f89025f07.slice/crio-78150070cbba23c4c0e4acab6798594c7b2c1bc5e2d090f1763a490eeeb4efea WatchSource:0}: Error finding container 78150070cbba23c4c0e4acab6798594c7b2c1bc5e2d090f1763a490eeeb4efea: Status 404 returned error can't find the container with id 78150070cbba23c4c0e4acab6798594c7b2c1bc5e2d090f1763a490eeeb4efea Nov 02 14:13:15.719863 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:15.719843 4866 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:13:16.288439 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:16.288350 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" event={"ID":"27f22147-746a-4848-bfe4-b31f89025f07","Type":"ContainerStarted","Data":"36460fdf7320caeea19e903cfb21aa905545ecc87a1f360ea1698c39c807e0ac"} Nov 02 14:13:16.288439 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:16.288406 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" event={"ID":"27f22147-746a-4848-bfe4-b31f89025f07","Type":"ContainerStarted","Data":"78150070cbba23c4c0e4acab6798594c7b2c1bc5e2d090f1763a490eeeb4efea"} Nov 02 14:13:16.288662 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:16.288461 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" Nov 02 14:13:16.332295 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:16.332234 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" podStartSLOduration=2.01432897 podStartE2EDuration="2.332220026s" totalImagesPullingTime="317.891056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:16.330002055 +0000 UTC m=+961.513849389" watchObservedRunningTime="2026-11-02 14:13:16.332220026 +0000 UTC m=+961.516067363" Nov 02 14:13:19.133873 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:19.133824 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx"] Nov 02 14:13:22.301555 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:22.301523 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" Nov 02 14:13:23.339039 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.339009 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m"] Nov 02 14:13:23.343196 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.343175 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.376577 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.376540 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m"] Nov 02 14:13:23.379598 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.379573 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8"] Nov 02 14:13:23.383560 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.383543 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.402502 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.402473 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8"] Nov 02 14:13:23.410330 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.410299 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.410490 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.410376 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.410490 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.410410 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.410490 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.410449 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-875bs\" (UniqueName: \"kubernetes.io/projected/8d0a11e0-7de7-4661-aac7-f785b2dc7777-kube-api-access-875bs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.410490 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.410476 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.410688 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.410498 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.410688 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.410550 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65cbb15a-d104-4b88-9147-290d212c6842-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.410688 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.410574 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65cbb15a-d104-4b88-9147-290d212c6842-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.410688 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.410609 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65cbb15a-d104-4b88-9147-290d212c6842-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.410811 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.410690 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.410811 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.410731 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.410811 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.410783 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.410904 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.410849 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.410937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.410924 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j94sq\" (UniqueName: \"kubernetes.io/projected/65cbb15a-d104-4b88-9147-290d212c6842-kube-api-access-j94sq\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.411011 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.410964 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.411065 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.411029 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.411065 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.411055 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.411170 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.411094 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.511753 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.511721 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.511976 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.511761 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-875bs\" (UniqueName: \"kubernetes.io/projected/8d0a11e0-7de7-4661-aac7-f785b2dc7777-kube-api-access-875bs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.511976 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.511784 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.511976 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.511803 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.511976 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.511855 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65cbb15a-d104-4b88-9147-290d212c6842-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.511976 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.511879 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65cbb15a-d104-4b88-9147-290d212c6842-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.511976 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.511916 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65cbb15a-d104-4b88-9147-290d212c6842-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.512335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.511985 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.512335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.512010 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.512335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.512051 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.512335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.512118 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.512335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.512186 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j94sq\" (UniqueName: \"kubernetes.io/projected/65cbb15a-d104-4b88-9147-290d212c6842-kube-api-access-j94sq\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.512335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.512211 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.512335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.512254 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.512335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.512256 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.512335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.512276 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.512335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.512307 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.512335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.512333 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.512902 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.512351 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.512902 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.512500 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.512902 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.512498 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.512902 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.512578 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.512902 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.512856 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.513192 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.512916 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.513192 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.513002 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65cbb15a-d104-4b88-9147-290d212c6842-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.513192 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.513124 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.513192 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.513151 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.513624 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.513594 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.514924 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.514875 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.515340 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.515318 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.515452 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.515376 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.515545 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.515525 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65cbb15a-d104-4b88-9147-290d212c6842-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.538257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.538203 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65cbb15a-d104-4b88-9147-290d212c6842-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.538432 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.538408 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j94sq\" (UniqueName: \"kubernetes.io/projected/65cbb15a-d104-4b88-9147-290d212c6842-kube-api-access-j94sq\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.538484 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.538459 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.538587 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.538569 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-875bs\" (UniqueName: \"kubernetes.io/projected/8d0a11e0-7de7-4661-aac7-f785b2dc7777-kube-api-access-875bs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.656718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.656636 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:23.700614 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.700579 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:23.981459 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.981385 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m"] Nov 02 14:13:23.982862 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:13:23.982834 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d0a11e0_7de7_4661_aac7_f785b2dc7777.slice/crio-8c7d0e70aa4df5880d7020d7d9f0b71474d11aec5dad25265d9cc4908b6f80c6 WatchSource:0}: Error finding container 8c7d0e70aa4df5880d7020d7d9f0b71474d11aec5dad25265d9cc4908b6f80c6: Status 404 returned error can't find the container with id 8c7d0e70aa4df5880d7020d7d9f0b71474d11aec5dad25265d9cc4908b6f80c6 Nov 02 14:13:23.985670 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.985629 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:13:23.985766 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.985690 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:13:23.985766 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:23.985719 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:13:24.025203 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:24.023474 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8"] Nov 02 14:13:24.027574 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:13:24.027547 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65cbb15a_d104_4b88_9147_290d212c6842.slice/crio-5c268f2e0cf5e6ea1b561ec8025e5e5817e74c0105d439fa7f02bca87740a653 WatchSource:0}: Error finding container 5c268f2e0cf5e6ea1b561ec8025e5e5817e74c0105d439fa7f02bca87740a653: Status 404 returned error can't find the container with id 5c268f2e0cf5e6ea1b561ec8025e5e5817e74c0105d439fa7f02bca87740a653 Nov 02 14:13:24.030773 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:24.030739 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:13:24.030881 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:24.030821 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:13:24.030881 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:24.030868 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:13:24.336754 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:24.336657 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" event={"ID":"65cbb15a-d104-4b88-9147-290d212c6842","Type":"ContainerStarted","Data":"d533a9420cde800e9a72a49b5c82a9a2b1b9f41195bba232db215b97b2055188"} Nov 02 14:13:24.336932 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:24.336767 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" event={"ID":"65cbb15a-d104-4b88-9147-290d212c6842","Type":"ContainerStarted","Data":"5c268f2e0cf5e6ea1b561ec8025e5e5817e74c0105d439fa7f02bca87740a653"} Nov 02 14:13:24.341987 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:24.338643 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" event={"ID":"8d0a11e0-7de7-4661-aac7-f785b2dc7777","Type":"ContainerStarted","Data":"e76a519d7b570438709f49cc02af0b25e971e090bf7d5842f897a6e8d291a2ce"} Nov 02 14:13:24.341987 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:24.338698 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" event={"ID":"8d0a11e0-7de7-4661-aac7-f785b2dc7777","Type":"ContainerStarted","Data":"8c7d0e70aa4df5880d7020d7d9f0b71474d11aec5dad25265d9cc4908b6f80c6"} Nov 02 14:13:24.385174 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:24.385120 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" podStartSLOduration=1.385104014 podStartE2EDuration="1.385104014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:24.381475141 +0000 UTC m=+969.565322482" watchObservedRunningTime="2026-11-02 14:13:24.385104014 +0000 UTC m=+969.568951418" Nov 02 14:13:24.429659 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:24.429602 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" podStartSLOduration=1.429588769 podStartE2EDuration="1.429588769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:24.427282529 +0000 UTC m=+969.611129866" watchObservedRunningTime="2026-11-02 14:13:24.429588769 +0000 UTC m=+969.613436105" Nov 02 14:13:24.657033 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:24.656935 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:24.700964 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:24.700913 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:25.659259 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:25.659226 4866 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 14:13:25.659646 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:25.659277 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" podUID="8d0a11e0-7de7-4661-aac7-f785b2dc7777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 14:13:25.718978 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:25.717806 4866 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 14:13:25.718978 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:25.717871 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" podUID="65cbb15a-d104-4b88-9147-290d212c6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 14:13:26.667009 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:26.666339 4866 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 14:13:26.667009 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:26.666397 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" podUID="8d0a11e0-7de7-4661-aac7-f785b2dc7777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 14:13:26.703973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:26.702959 4866 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 14:13:26.703973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:26.703015 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" podUID="65cbb15a-d104-4b88-9147-290d212c6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 14:13:27.658973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:27.658394 4866 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 14:13:27.658973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:27.658459 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" podUID="8d0a11e0-7de7-4661-aac7-f785b2dc7777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 14:13:27.710971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:27.706759 4866 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 14:13:27.710971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:27.706816 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" podUID="65cbb15a-d104-4b88-9147-290d212c6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 14:13:28.646973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:28.646079 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" podUID="9753ddaa-3899-46ea-952f-5f03d76037c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:28.664983 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:28.659533 4866 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 14:13:28.664983 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:28.659604 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" podUID="8d0a11e0-7de7-4661-aac7-f785b2dc7777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 14:13:28.703972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:28.703447 4866 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 14:13:28.703972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:28.703526 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" podUID="65cbb15a-d104-4b88-9147-290d212c6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 14:13:28.805975 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:28.805033 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" podUID="d7037eab-3f5f-419b-94f6-784ff7331925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:28.806503 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:28.806235 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" podUID="df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:28.901029 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:28.900032 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" podUID="6d89a28a-7f87-4924-93c3-48ae7d11e56e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:29.448971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:29.448088 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" podUID="0129df2e-9d21-45f7-9aa2-53777ce753b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:29.596764 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:29.595070 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" podUID="8b7821c4-51bb-4d23-ae57-c2f4803cbacb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 14:13:29.757318 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:29.652065 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" podUID="9753ddaa-3899-46ea-952f-5f03d76037c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:29.757318 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:29.652707 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:13:29.795973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:29.769344 4866 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 14:13:29.795973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:29.769414 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" podUID="65cbb15a-d104-4b88-9147-290d212c6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 14:13:29.795973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:29.769435 4866 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 14:13:29.795973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:29.769460 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" podUID="8d0a11e0-7de7-4661-aac7-f785b2dc7777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 14:13:29.871976 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:29.814042 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" podUID="d7037eab-3f5f-419b-94f6-784ff7331925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:29.871976 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:29.814548 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:13:29.871976 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:29.816380 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" podUID="df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:29.871976 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:29.816703 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:13:29.966442 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:29.966391 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" podUID="6d89a28a-7f87-4924-93c3-48ae7d11e56e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:29.967111 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:29.967088 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:13:30.114030 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:30.103789 4866 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" podUID="9753ddaa-3899-46ea-952f-5f03d76037c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:30.455773 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:30.452057 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" podUID="0129df2e-9d21-45f7-9aa2-53777ce753b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:30.455773 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:30.453637 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:13:30.488515 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:30.488471 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:13:30.584094 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:30.583630 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:13:30.584427 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:30.584406 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:13:30.675798 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:30.675762 4866 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 14:13:30.677566 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:30.677416 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" podUID="8d0a11e0-7de7-4661-aac7-f785b2dc7777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 14:13:30.685396 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:30.679174 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:13:30.688076 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:30.688054 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:13:30.711292 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:30.711206 4866 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 14:13:30.711537 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:30.711509 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" podUID="65cbb15a-d104-4b88-9147-290d212c6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 14:13:31.657583 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:31.657535 4866 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 14:13:31.658142 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:31.657612 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" podUID="8d0a11e0-7de7-4661-aac7-f785b2dc7777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 14:13:31.701119 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:31.701085 4866 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 14:13:31.701321 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:31.701136 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" podUID="65cbb15a-d104-4b88-9147-290d212c6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 14:13:32.933617 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:32.933540 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:32.934746 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:32.934719 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:32.936461 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:32.936443 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:13:33.085402 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:33.084071 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:33.085402 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:33.085251 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:33.086371 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:33.086350 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:13:34.154988 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.154930 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" podUID="8b7821c4-51bb-4d23-ae57-c2f4803cbacb" containerName="main" containerID="cri-o://22ab9adda97879bee8155a9a80a754d8aba800d50da860d8737172cbecd98838" gracePeriod=45 Nov 02 14:13:34.426972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.426932 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:13:34.511723 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.511686 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-model-cache\") pod \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " Nov 02 14:13:34.511929 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.511752 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-home\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-home\") pod \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " Nov 02 14:13:34.511929 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.511850 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-tls-certs\") pod \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " Nov 02 14:13:34.511929 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.511879 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-kube-api-access-d9xrj\" (UniqueName: \"kubernetes.io/projected/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-kube-api-access-d9xrj\") pod \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " Nov 02 14:13:34.511929 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.511924 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-dshm\") pod \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " Nov 02 14:13:34.512179 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.512092 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-tmp-dir\") pod \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\" (UID: \"8b7821c4-51bb-4d23-ae57-c2f4803cbacb\") " Nov 02 14:13:34.512179 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.512102 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-home" pod "8b7821c4-51bb-4d23-ae57-c2f4803cbacb" (UID: "8b7821c4-51bb-4d23-ae57-c2f4803cbacb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:13:34.512691 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.512641 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-tmp-dir" pod "8b7821c4-51bb-4d23-ae57-c2f4803cbacb" (UID: "8b7821c4-51bb-4d23-ae57-c2f4803cbacb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:13:34.512691 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.512650 4866 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-home\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:13:34.514673 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.512018 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-model-cache" pod "8b7821c4-51bb-4d23-ae57-c2f4803cbacb" (UID: "8b7821c4-51bb-4d23-ae57-c2f4803cbacb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:13:34.514868 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.514842 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-dshm" pod "8b7821c4-51bb-4d23-ae57-c2f4803cbacb" (UID: "8b7821c4-51bb-4d23-ae57-c2f4803cbacb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:13:34.514995 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.514881 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-kube-api-access-d9xrj" pod "8b7821c4-51bb-4d23-ae57-c2f4803cbacb" (UID: "8b7821c4-51bb-4d23-ae57-c2f4803cbacb"). InnerVolumeSpecName "kube-api-access-d9xrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:13:34.515273 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.515253 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-tls-certs" pod "8b7821c4-51bb-4d23-ae57-c2f4803cbacb" (UID: "8b7821c4-51bb-4d23-ae57-c2f4803cbacb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:13:34.582542 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.582509 4866 generic.go:356] "Generic (PLEG): container finished" podID="8b7821c4-51bb-4d23-ae57-c2f4803cbacb" containerID="22ab9adda97879bee8155a9a80a754d8aba800d50da860d8737172cbecd98838" exitCode=0 Nov 02 14:13:34.582734 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.582554 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" Nov 02 14:13:34.582734 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.582586 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" event={"ID":"8b7821c4-51bb-4d23-ae57-c2f4803cbacb","Type":"ContainerDied","Data":"22ab9adda97879bee8155a9a80a754d8aba800d50da860d8737172cbecd98838"} Nov 02 14:13:34.582734 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.582627 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx" event={"ID":"8b7821c4-51bb-4d23-ae57-c2f4803cbacb","Type":"ContainerDied","Data":"7c0721489eac1ceb1cc89e606d18cbaa9e1e35f12e7b1a7a6d7b7854cb97131d"} Nov 02 14:13:34.582734 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.582644 4866 scope.go:122] "RemoveContainer" containerID="22ab9adda97879bee8155a9a80a754d8aba800d50da860d8737172cbecd98838" Nov 02 14:13:34.600799 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.600778 4866 scope.go:122] "RemoveContainer" containerID="22ab9adda97879bee8155a9a80a754d8aba800d50da860d8737172cbecd98838" Nov 02 14:13:34.601121 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:13:34.601103 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ab9adda97879bee8155a9a80a754d8aba800d50da860d8737172cbecd98838\": container with ID starting with 22ab9adda97879bee8155a9a80a754d8aba800d50da860d8737172cbecd98838 not found: ID does not exist" containerID="22ab9adda97879bee8155a9a80a754d8aba800d50da860d8737172cbecd98838" Nov 02 14:13:34.601186 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.601131 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ab9adda97879bee8155a9a80a754d8aba800d50da860d8737172cbecd98838"} err="failed to get container status \"22ab9adda97879bee8155a9a80a754d8aba800d50da860d8737172cbecd98838\": rpc error: code = NotFound desc = could not find container \"22ab9adda97879bee8155a9a80a754d8aba800d50da860d8737172cbecd98838\": container with ID starting with 22ab9adda97879bee8155a9a80a754d8aba800d50da860d8737172cbecd98838 not found: ID does not exist" Nov 02 14:13:34.613792 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.613666 4866 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-tls-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:13:34.613792 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.613699 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9xrj\" (UniqueName: \"kubernetes.io/projected/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-kube-api-access-d9xrj\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:13:34.613792 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.613732 4866 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-dshm\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:13:34.613792 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.613767 4866 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-tmp-dir\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:13:34.613792 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.613781 4866 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7821c4-51bb-4d23-ae57-c2f4803cbacb-model-cache\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:13:34.635758 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.635723 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx"] Nov 02 14:13:34.641296 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.641270 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-854b9e/test-model-854b9e-kserve-7f77f88ccc-84qlx"] Nov 02 14:13:34.947054 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:34.947022 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b7821c4-51bb-4d23-ae57-c2f4803cbacb" path="/var/lib/kubelet/pods/8b7821c4-51bb-4d23-ae57-c2f4803cbacb/volumes" Nov 02 14:13:36.055238 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:13:36.055144 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:13:51.176367 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:51.176332 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59"] Nov 02 14:13:56.177262 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.177212 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" podUID="50eee8c0-0f94-4633-9732-bfdd1d6c9f68" containerName="payload-processing" containerID="cri-o://b5994338f13637bfb380b38dbe6e928c4b9b618e37f2bc4b1f88264fd5988a34" gracePeriod=25 Nov 02 14:13:56.455014 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.454987 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" Nov 02 14:13:56.500720 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.500668 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-routing-overlay-volume\") pod \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\" (UID: \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\") " Nov 02 14:13:56.500902 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.500818 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-kube-api-access-hg7xc\" (UniqueName: \"kubernetes.io/projected/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-kube-api-access-hg7xc\") pod \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\" (UID: \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\") " Nov 02 14:13:56.500902 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.500849 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-provider-credentials\" (UniqueName: \"kubernetes.io/projected/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-provider-credentials\") pod \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\" (UID: \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\") " Nov 02 14:13:56.501069 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.501012 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-plugins-config-volume\") pod \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\" (UID: \"50eee8c0-0f94-4633-9732-bfdd1d6c9f68\") " Nov 02 14:13:56.501433 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.501381 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-routing-overlay-volume" pod "50eee8c0-0f94-4633-9732-bfdd1d6c9f68" (UID: "50eee8c0-0f94-4633-9732-bfdd1d6c9f68"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:13:56.501698 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.501560 4866 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-routing-overlay-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:13:56.501698 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.501686 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-plugins-config-volume" pod "50eee8c0-0f94-4633-9732-bfdd1d6c9f68" (UID: "50eee8c0-0f94-4633-9732-bfdd1d6c9f68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:13:56.504384 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.504337 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-provider-credentials" pod "50eee8c0-0f94-4633-9732-bfdd1d6c9f68" (UID: "50eee8c0-0f94-4633-9732-bfdd1d6c9f68"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:13:56.509471 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.509434 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-kube-api-access-hg7xc" pod "50eee8c0-0f94-4633-9732-bfdd1d6c9f68" (UID: "50eee8c0-0f94-4633-9732-bfdd1d6c9f68"). InnerVolumeSpecName "kube-api-access-hg7xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:13:56.602840 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.602807 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg7xc\" (UniqueName: \"kubernetes.io/projected/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-kube-api-access-hg7xc\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:13:56.602840 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.602836 4866 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-provider-credentials\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:13:56.602840 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.602846 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eee8c0-0f94-4633-9732-bfdd1d6c9f68-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:13:56.794930 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.794900 4866 generic.go:356] "Generic (PLEG): container finished" podID="50eee8c0-0f94-4633-9732-bfdd1d6c9f68" containerID="b5994338f13637bfb380b38dbe6e928c4b9b618e37f2bc4b1f88264fd5988a34" exitCode=0 Nov 02 14:13:56.794930 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.794937 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" event={"ID":"50eee8c0-0f94-4633-9732-bfdd1d6c9f68","Type":"ContainerDied","Data":"b5994338f13637bfb380b38dbe6e928c4b9b618e37f2bc4b1f88264fd5988a34"} Nov 02 14:13:56.795146 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.794956 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" Nov 02 14:13:56.795146 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.794994 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59" event={"ID":"50eee8c0-0f94-4633-9732-bfdd1d6c9f68","Type":"ContainerDied","Data":"c94135adfbd16ff4c3b324856186090f644623d4f969a7712c574a9e667bd582"} Nov 02 14:13:56.795146 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.795016 4866 scope.go:122] "RemoveContainer" containerID="b5994338f13637bfb380b38dbe6e928c4b9b618e37f2bc4b1f88264fd5988a34" Nov 02 14:13:56.806420 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.806396 4866 scope.go:122] "RemoveContainer" containerID="b5994338f13637bfb380b38dbe6e928c4b9b618e37f2bc4b1f88264fd5988a34" Nov 02 14:13:56.806771 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:13:56.806726 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5994338f13637bfb380b38dbe6e928c4b9b618e37f2bc4b1f88264fd5988a34\": container with ID starting with b5994338f13637bfb380b38dbe6e928c4b9b618e37f2bc4b1f88264fd5988a34 not found: ID does not exist" containerID="b5994338f13637bfb380b38dbe6e928c4b9b618e37f2bc4b1f88264fd5988a34" Nov 02 14:13:56.806862 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.806779 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5994338f13637bfb380b38dbe6e928c4b9b618e37f2bc4b1f88264fd5988a34"} err="failed to get container status \"b5994338f13637bfb380b38dbe6e928c4b9b618e37f2bc4b1f88264fd5988a34\": rpc error: code = NotFound desc = could not find container \"b5994338f13637bfb380b38dbe6e928c4b9b618e37f2bc4b1f88264fd5988a34\": container with ID starting with b5994338f13637bfb380b38dbe6e928c4b9b618e37f2bc4b1f88264fd5988a34 not found: ID does not exist" Nov 02 14:13:56.825552 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.825522 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n"] Nov 02 14:13:56.826415 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.826391 4866 cpu_manager.go:414] "containerMap: removing container" podUID="50eee8c0-0f94-4633-9732-bfdd1d6c9f68" containerName="payload-processing" Nov 02 14:13:56.826538 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.826416 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50eee8c0-0f94-4633-9732-bfdd1d6c9f68" containerName="payload-processing" Nov 02 14:13:56.826538 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.826438 4866 cpu_manager.go:414] "containerMap: removing container" podUID="8b7821c4-51bb-4d23-ae57-c2f4803cbacb" containerName="main" Nov 02 14:13:56.826538 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.826448 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b7821c4-51bb-4d23-ae57-c2f4803cbacb" containerName="main" Nov 02 14:13:56.826700 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.826630 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="50eee8c0-0f94-4633-9732-bfdd1d6c9f68" containerName="payload-processing" Nov 02 14:13:56.826700 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.826645 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b7821c4-51bb-4d23-ae57-c2f4803cbacb" containerName="main" Nov 02 14:13:56.831753 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.831731 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:56.848941 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.848913 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4be645\"/\"distinct-2-w2-4be645-kserve-self-signed-certs\"" Nov 02 14:13:56.850898 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.850870 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n"] Nov 02 14:13:56.857064 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.857033 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59"] Nov 02 14:13:56.860280 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.860254 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdwkq59"] Nov 02 14:13:56.905446 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.905414 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7q4h\" (UniqueName: \"kubernetes.io/projected/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-kube-api-access-r7q4h\") pod \"distinct-2-w2-4be645-kserve-66df89565d-xl64n\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:56.905599 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.905491 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-tmp-dir\") pod \"distinct-2-w2-4be645-kserve-66df89565d-xl64n\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:56.905599 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.905559 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-home\") pod \"distinct-2-w2-4be645-kserve-66df89565d-xl64n\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:56.905669 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.905609 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-tls-certs\") pod \"distinct-2-w2-4be645-kserve-66df89565d-xl64n\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:56.905669 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.905650 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-dshm\") pod \"distinct-2-w2-4be645-kserve-66df89565d-xl64n\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:56.905785 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.905773 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-model-cache\") pod \"distinct-2-w2-4be645-kserve-66df89565d-xl64n\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:56.946775 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:56.946741 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50eee8c0-0f94-4633-9732-bfdd1d6c9f68" path="/var/lib/kubelet/pods/50eee8c0-0f94-4633-9732-bfdd1d6c9f68/volumes" Nov 02 14:13:57.007289 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:57.007252 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7q4h\" (UniqueName: \"kubernetes.io/projected/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-kube-api-access-r7q4h\") pod \"distinct-2-w2-4be645-kserve-66df89565d-xl64n\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:57.007477 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:57.007382 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-tmp-dir\") pod \"distinct-2-w2-4be645-kserve-66df89565d-xl64n\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:57.007477 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:57.007451 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-home\") pod \"distinct-2-w2-4be645-kserve-66df89565d-xl64n\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:57.007477 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:57.007475 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-tls-certs\") pod \"distinct-2-w2-4be645-kserve-66df89565d-xl64n\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:57.008140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:57.007521 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-dshm\") pod \"distinct-2-w2-4be645-kserve-66df89565d-xl64n\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:57.008140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:57.007659 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-model-cache\") pod \"distinct-2-w2-4be645-kserve-66df89565d-xl64n\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:57.008140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:57.007902 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-home\") pod \"distinct-2-w2-4be645-kserve-66df89565d-xl64n\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:57.008140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:57.008108 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-model-cache\") pod \"distinct-2-w2-4be645-kserve-66df89565d-xl64n\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:57.008387 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:57.008185 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-tmp-dir\") pod \"distinct-2-w2-4be645-kserve-66df89565d-xl64n\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:57.010763 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:57.010732 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-tls-certs\") pod \"distinct-2-w2-4be645-kserve-66df89565d-xl64n\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:57.011073 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:57.011050 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-dshm\") pod \"distinct-2-w2-4be645-kserve-66df89565d-xl64n\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:57.035762 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:57.035727 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7q4h\" (UniqueName: \"kubernetes.io/projected/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-kube-api-access-r7q4h\") pod \"distinct-2-w2-4be645-kserve-66df89565d-xl64n\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:57.148969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:57.148866 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:13:57.508057 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:57.508034 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n"] Nov 02 14:13:57.510507 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:13:57.510480 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f9dd65a_ec89_40fb_8fd5_4bae69a0ee7e.slice/crio-e4e81592ab000ac14610cddd5b97e81e4df524df4705bf4abeea0caded973c4f WatchSource:0}: Error finding container e4e81592ab000ac14610cddd5b97e81e4df524df4705bf4abeea0caded973c4f: Status 404 returned error can't find the container with id e4e81592ab000ac14610cddd5b97e81e4df524df4705bf4abeea0caded973c4f Nov 02 14:13:57.803217 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:57.803125 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" event={"ID":"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e","Type":"ContainerStarted","Data":"ebd19e9adf5c442dc1691c9810a4fcd70d177c5fe7d1b3b6c3cec3262b7dc1e3"} Nov 02 14:13:57.803217 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:57.803173 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" event={"ID":"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e","Type":"ContainerStarted","Data":"e4e81592ab000ac14610cddd5b97e81e4df524df4705bf4abeea0caded973c4f"} Nov 02 14:13:57.854652 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:57.854589 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" podStartSLOduration=1.8545734249999999 podStartE2EDuration="1.854573425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:57.8532691 +0000 UTC m=+1003.037116437" watchObservedRunningTime="2026-11-02 14:13:57.854573425 +0000 UTC m=+1003.038420761" Nov 02 14:13:58.150050 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:13:58.149933 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:14:02.637836 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.637802 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf"] Nov 02 14:14:02.642077 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.642051 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.659810 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.659778 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 14:14:02.659994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.659829 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xcrsw\"" Nov 02 14:14:02.662338 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.662318 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 14:14:02.671790 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.671758 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 14:14:02.675591 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.675558 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf"] Nov 02 14:14:02.703024 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.702772 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07de4ac7-3fc8-4416-9464-5ec3144a9e03-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.703024 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.702935 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.703024 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.702998 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.703330 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.703225 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k62dn\" (UniqueName: \"kubernetes.io/projected/07de4ac7-3fc8-4416-9464-5ec3144a9e03-kube-api-access-k62dn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.703330 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.703269 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.703444 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.703384 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.703500 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.703454 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.804167 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.804117 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.804357 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.804253 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k62dn\" (UniqueName: \"kubernetes.io/projected/07de4ac7-3fc8-4416-9464-5ec3144a9e03-kube-api-access-k62dn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.804357 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.804278 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.804357 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.804317 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.804497 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.804376 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.804497 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.804408 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07de4ac7-3fc8-4416-9464-5ec3144a9e03-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.804586 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.804497 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.805174 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.805141 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.806835 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.806814 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07de4ac7-3fc8-4416-9464-5ec3144a9e03-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.807068 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.807048 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.807282 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.807263 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.807320 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.807274 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.855273 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.855235 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.855559 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.855535 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k62dn\" (UniqueName: \"kubernetes.io/projected/07de4ac7-3fc8-4416-9464-5ec3144a9e03-kube-api-access-k62dn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdlmf\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:02.957896 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:02.957866 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:03.282016 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:03.281993 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf"] Nov 02 14:14:03.283457 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:14:03.283430 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07de4ac7_3fc8_4416_9464_5ec3144a9e03.slice/crio-459c3af81bac6aeb006b448eaf2f2c89d750dd98b20d1ee7f501c1aa42034f89 WatchSource:0}: Error finding container 459c3af81bac6aeb006b448eaf2f2c89d750dd98b20d1ee7f501c1aa42034f89: Status 404 returned error can't find the container with id 459c3af81bac6aeb006b448eaf2f2c89d750dd98b20d1ee7f501c1aa42034f89 Nov 02 14:14:03.285869 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:03.285840 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:14:03.285933 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:03.285901 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:14:03.844858 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:03.844818 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" event={"ID":"07de4ac7-3fc8-4416-9464-5ec3144a9e03","Type":"ContainerStarted","Data":"72e8f2fa4674da6782cdc713d37ebf48efacb98391fdc51727b532447dfd0842"} Nov 02 14:14:03.845261 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:03.844876 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" event={"ID":"07de4ac7-3fc8-4416-9464-5ec3144a9e03","Type":"ContainerStarted","Data":"459c3af81bac6aeb006b448eaf2f2c89d750dd98b20d1ee7f501c1aa42034f89"} Nov 02 14:14:03.845261 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:03.844924 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:04.853960 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:04.853921 4866 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rdlmf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:14:04.854342 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:04.854006 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" podUID="07de4ac7-3fc8-4416-9464-5ec3144a9e03" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:14:06.009964 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:14:06.009903 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:14:07.149117 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:07.149083 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:14:07.161851 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:07.161818 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:14:07.208936 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:07.208879 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" podStartSLOduration=5.208863522 podStartE2EDuration="5.208863522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:03.912912512 +0000 UTC m=+1009.096759849" watchObservedRunningTime="2026-11-02 14:14:07.208863522 +0000 UTC m=+1012.392710866" Nov 02 14:14:07.852660 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:07.852625 4866 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rdlmf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:14:07.852820 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:07.852686 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" podUID="07de4ac7-3fc8-4416-9464-5ec3144a9e03" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:14:07.891904 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:07.891871 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:14:08.308405 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:08.308369 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts"] Nov 02 14:14:10.853142 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:10.853104 4866 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rdlmf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:14:10.853759 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:10.853728 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" podUID="07de4ac7-3fc8-4416-9464-5ec3144a9e03" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:14:13.309346 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:13.309302 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" podUID="8810f722-0ee3-41e5-a77a-ef9e5da7eeea" containerName="payload-processing" containerID="cri-o://92afd22caae634e7c54bd19220a783c3a49c1b667b16589276093d2899faf60e" gracePeriod=25 Nov 02 14:14:13.852351 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:13.852318 4866 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rdlmf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:14:13.852525 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:13.852384 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" podUID="07de4ac7-3fc8-4416-9464-5ec3144a9e03" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:14:16.854992 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:16.854443 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:14:21.690818 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:21.690780 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" podUID="8810f722-0ee3-41e5-a77a-ef9e5da7eeea" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:14:29.885143 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:29.885034 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" podUID="4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:14:30.071102 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:30.071048 4866 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" podUID="9753ddaa-3899-46ea-952f-5f03d76037c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:14:30.493007 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:30.492010 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" podUID="6d89a28a-7f87-4924-93c3-48ae7d11e56e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded" Nov 02 14:14:30.610971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:30.594055 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" podUID="d7037eab-3f5f-419b-94f6-784ff7331925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:14:30.758159 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:30.758046 4866 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" podUID="d7037eab-3f5f-419b-94f6-784ff7331925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:14:31.690703 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:31.690656 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" podUID="8810f722-0ee3-41e5-a77a-ef9e5da7eeea" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:14:33.580061 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:33.580039 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" Nov 02 14:14:33.679864 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:33.679821 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8810f722-0ee3-41e5-a77a-ef9e5da7eeea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8810f722-0ee3-41e5-a77a-ef9e5da7eeea-plugins-config-volume\") pod \"8810f722-0ee3-41e5-a77a-ef9e5da7eeea\" (UID: \"8810f722-0ee3-41e5-a77a-ef9e5da7eeea\") " Nov 02 14:14:33.680062 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:33.680022 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8810f722-0ee3-41e5-a77a-ef9e5da7eeea-kube-api-access-dgj8r\" (UniqueName: \"kubernetes.io/projected/8810f722-0ee3-41e5-a77a-ef9e5da7eeea-kube-api-access-dgj8r\") pod \"8810f722-0ee3-41e5-a77a-ef9e5da7eeea\" (UID: \"8810f722-0ee3-41e5-a77a-ef9e5da7eeea\") " Nov 02 14:14:33.680294 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:33.680262 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8810f722-0ee3-41e5-a77a-ef9e5da7eeea-plugins-config-volume" pod "8810f722-0ee3-41e5-a77a-ef9e5da7eeea" (UID: "8810f722-0ee3-41e5-a77a-ef9e5da7eeea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:14:33.680609 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:33.680588 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8810f722-0ee3-41e5-a77a-ef9e5da7eeea-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:14:33.682488 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:33.682465 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8810f722-0ee3-41e5-a77a-ef9e5da7eeea-kube-api-access-dgj8r" pod "8810f722-0ee3-41e5-a77a-ef9e5da7eeea" (UID: "8810f722-0ee3-41e5-a77a-ef9e5da7eeea"). InnerVolumeSpecName "kube-api-access-dgj8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:14:33.781522 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:33.781488 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgj8r\" (UniqueName: \"kubernetes.io/projected/8810f722-0ee3-41e5-a77a-ef9e5da7eeea-kube-api-access-dgj8r\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:14:34.186080 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:34.186041 4866 generic.go:356] "Generic (PLEG): container finished" podID="8810f722-0ee3-41e5-a77a-ef9e5da7eeea" containerID="92afd22caae634e7c54bd19220a783c3a49c1b667b16589276093d2899faf60e" exitCode=0 Nov 02 14:14:34.186291 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:34.186083 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" Nov 02 14:14:34.186291 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:34.186131 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" event={"ID":"8810f722-0ee3-41e5-a77a-ef9e5da7eeea","Type":"ContainerDied","Data":"92afd22caae634e7c54bd19220a783c3a49c1b667b16589276093d2899faf60e"} Nov 02 14:14:34.186291 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:34.186183 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts" event={"ID":"8810f722-0ee3-41e5-a77a-ef9e5da7eeea","Type":"ContainerDied","Data":"e8472fd666991b43b0e11a27d77fb119db498f505cd2bff1f5782002c845f5c2"} Nov 02 14:14:34.186291 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:34.186214 4866 scope.go:122] "RemoveContainer" containerID="92afd22caae634e7c54bd19220a783c3a49c1b667b16589276093d2899faf60e" Nov 02 14:14:34.200054 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:34.200031 4866 scope.go:122] "RemoveContainer" containerID="92afd22caae634e7c54bd19220a783c3a49c1b667b16589276093d2899faf60e" Nov 02 14:14:34.200374 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:14:34.200350 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92afd22caae634e7c54bd19220a783c3a49c1b667b16589276093d2899faf60e\": container with ID starting with 92afd22caae634e7c54bd19220a783c3a49c1b667b16589276093d2899faf60e not found: ID does not exist" containerID="92afd22caae634e7c54bd19220a783c3a49c1b667b16589276093d2899faf60e" Nov 02 14:14:34.200473 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:34.200382 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92afd22caae634e7c54bd19220a783c3a49c1b667b16589276093d2899faf60e"} err="failed to get container status \"92afd22caae634e7c54bd19220a783c3a49c1b667b16589276093d2899faf60e\": rpc error: code = NotFound desc = could not find container \"92afd22caae634e7c54bd19220a783c3a49c1b667b16589276093d2899faf60e\": container with ID starting with 92afd22caae634e7c54bd19220a783c3a49c1b667b16589276093d2899faf60e not found: ID does not exist" Nov 02 14:14:34.240147 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:34.240117 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts"] Nov 02 14:14:34.251282 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:34.251252 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-854b9e-56687854c7-2m9ts"] Nov 02 14:14:34.946170 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:14:34.946136 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8810f722-0ee3-41e5-a77a-ef9e5da7eeea" path="/var/lib/kubelet/pods/8810f722-0ee3-41e5-a77a-ef9e5da7eeea/volumes" Nov 02 14:14:36.009505 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:14:36.009467 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:15:02.247745 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:02.247707 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn"] Nov 02 14:15:02.248290 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:02.248276 4866 cpu_manager.go:414] "containerMap: removing container" podUID="8810f722-0ee3-41e5-a77a-ef9e5da7eeea" containerName="payload-processing" Nov 02 14:15:02.248333 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:02.248290 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8810f722-0ee3-41e5-a77a-ef9e5da7eeea" containerName="payload-processing" Nov 02 14:15:02.248377 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:02.248369 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="8810f722-0ee3-41e5-a77a-ef9e5da7eeea" containerName="payload-processing" Nov 02 14:15:02.253187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:02.253169 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn" Nov 02 14:15:02.274110 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:02.274077 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn"] Nov 02 14:15:02.281964 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:02.281923 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:15:02.282195 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:02.282163 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:15:02.319566 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:02.319519 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c90642e-dcd1-43af-b612-243f4af10796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1b9b41b-664bn\" (UID: \"9c90642e-dcd1-43af-b612-243f4af10796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn" Nov 02 14:15:02.319746 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:02.319711 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c90642e-dcd1-43af-b612-243f4af10796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1b9b41b-664bn\" (UID: \"9c90642e-dcd1-43af-b612-243f4af10796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn" Nov 02 14:15:02.319809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:02.319768 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78j5j\" (UniqueName: \"kubernetes.io/projected/9c90642e-dcd1-43af-b612-243f4af10796-kube-api-access-78j5j\") pod \"maas-api-revoke-sub-f1b9b41b-664bn\" (UID: \"9c90642e-dcd1-43af-b612-243f4af10796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn" Nov 02 14:15:02.420573 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:02.420535 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c90642e-dcd1-43af-b612-243f4af10796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1b9b41b-664bn\" (UID: \"9c90642e-dcd1-43af-b612-243f4af10796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn" Nov 02 14:15:02.420746 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:02.420638 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c90642e-dcd1-43af-b612-243f4af10796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1b9b41b-664bn\" (UID: \"9c90642e-dcd1-43af-b612-243f4af10796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn" Nov 02 14:15:02.420746 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:02.420674 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78j5j\" (UniqueName: \"kubernetes.io/projected/9c90642e-dcd1-43af-b612-243f4af10796-kube-api-access-78j5j\") pod \"maas-api-revoke-sub-f1b9b41b-664bn\" (UID: \"9c90642e-dcd1-43af-b612-243f4af10796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn" Nov 02 14:15:02.421245 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:02.421225 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c90642e-dcd1-43af-b612-243f4af10796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1b9b41b-664bn\" (UID: \"9c90642e-dcd1-43af-b612-243f4af10796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn" Nov 02 14:15:02.423140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:02.423114 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c90642e-dcd1-43af-b612-243f4af10796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1b9b41b-664bn\" (UID: \"9c90642e-dcd1-43af-b612-243f4af10796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn" Nov 02 14:15:02.444963 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:02.444923 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78j5j\" (UniqueName: \"kubernetes.io/projected/9c90642e-dcd1-43af-b612-243f4af10796-kube-api-access-78j5j\") pod \"maas-api-revoke-sub-f1b9b41b-664bn\" (UID: \"9c90642e-dcd1-43af-b612-243f4af10796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn" Nov 02 14:15:02.567196 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:02.567110 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn" Nov 02 14:15:02.874111 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:02.873957 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn"] Nov 02 14:15:02.879410 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:15:02.879377 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c90642e_dcd1_43af_b612_243f4af10796.slice/crio-0fae613d3f817c9d048e06e8e7c42b5e51b15f232606873f6eb142834074f513 WatchSource:0}: Error finding container 0fae613d3f817c9d048e06e8e7c42b5e51b15f232606873f6eb142834074f513: Status 404 returned error can't find the container with id 0fae613d3f817c9d048e06e8e7c42b5e51b15f232606873f6eb142834074f513 Nov 02 14:15:03.409936 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:03.409892 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn" event={"ID":"9c90642e-dcd1-43af-b612-243f4af10796","Type":"ContainerStarted","Data":"0fae613d3f817c9d048e06e8e7c42b5e51b15f232606873f6eb142834074f513"} Nov 02 14:15:06.016704 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:15:06.016663 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:15:07.440031 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:07.440004 4866 generic.go:356] "Generic (PLEG): container finished" podID="9c90642e-dcd1-43af-b612-243f4af10796" containerID="4bb0d156829de41fcaff8d853ef2b92cb9602bb46292f59cf135bf9bbd9798e3" exitCode=0 Nov 02 14:15:07.440448 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:07.440094 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn" event={"ID":"9c90642e-dcd1-43af-b612-243f4af10796","Type":"ContainerDied","Data":"4bb0d156829de41fcaff8d853ef2b92cb9602bb46292f59cf135bf9bbd9798e3"} Nov 02 14:15:08.606118 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:08.606092 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn" Nov 02 14:15:08.756907 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:08.756818 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c90642e-dcd1-43af-b612-243f4af10796-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c90642e-dcd1-43af-b612-243f4af10796-maas-api-openshift-ca\") pod \"9c90642e-dcd1-43af-b612-243f4af10796\" (UID: \"9c90642e-dcd1-43af-b612-243f4af10796\") " Nov 02 14:15:08.757090 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:08.756921 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c90642e-dcd1-43af-b612-243f4af10796-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c90642e-dcd1-43af-b612-243f4af10796-maas-api-xks-ca\") pod \"9c90642e-dcd1-43af-b612-243f4af10796\" (UID: \"9c90642e-dcd1-43af-b612-243f4af10796\") " Nov 02 14:15:08.757090 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:08.757029 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c90642e-dcd1-43af-b612-243f4af10796-kube-api-access-78j5j\" (UniqueName: \"kubernetes.io/projected/9c90642e-dcd1-43af-b612-243f4af10796-kube-api-access-78j5j\") pod \"9c90642e-dcd1-43af-b612-243f4af10796\" (UID: \"9c90642e-dcd1-43af-b612-243f4af10796\") " Nov 02 14:15:08.757277 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:08.757240 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c90642e-dcd1-43af-b612-243f4af10796-maas-api-openshift-ca" pod "9c90642e-dcd1-43af-b612-243f4af10796" (UID: "9c90642e-dcd1-43af-b612-243f4af10796"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:15:08.757589 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:08.757569 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c90642e-dcd1-43af-b612-243f4af10796-maas-api-openshift-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:15:08.759107 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:08.759086 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c90642e-dcd1-43af-b612-243f4af10796-maas-api-xks-ca" pod "9c90642e-dcd1-43af-b612-243f4af10796" (UID: "9c90642e-dcd1-43af-b612-243f4af10796"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:15:08.759107 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:08.759095 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c90642e-dcd1-43af-b612-243f4af10796-kube-api-access-78j5j" pod "9c90642e-dcd1-43af-b612-243f4af10796" (UID: "9c90642e-dcd1-43af-b612-243f4af10796"). InnerVolumeSpecName "kube-api-access-78j5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:15:08.858405 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:08.858372 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c90642e-dcd1-43af-b612-243f4af10796-maas-api-xks-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:15:08.858405 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:08.858399 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78j5j\" (UniqueName: \"kubernetes.io/projected/9c90642e-dcd1-43af-b612-243f4af10796-kube-api-access-78j5j\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:15:09.462114 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:09.462088 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn" Nov 02 14:15:09.462292 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:09.462084 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn" event={"ID":"9c90642e-dcd1-43af-b612-243f4af10796","Type":"ContainerDied","Data":"0fae613d3f817c9d048e06e8e7c42b5e51b15f232606873f6eb142834074f513"} Nov 02 14:15:09.462292 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:09.462214 4866 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fae613d3f817c9d048e06e8e7c42b5e51b15f232606873f6eb142834074f513" Nov 02 14:15:28.521262 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:28.521213 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" podUID="6d89a28a-7f87-4924-93c3-48ae7d11e56e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:15:28.692095 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:28.692039 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" podUID="df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 02 14:15:28.692854 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:28.692380 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" podUID="0129df2e-9d21-45f7-9aa2-53777ce753b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:15:28.884194 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:28.884058 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" podUID="4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:15:29.530969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:29.525345 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" podUID="6d89a28a-7f87-4924-93c3-48ae7d11e56e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:15:29.530969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:29.526302 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:15:29.603505 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:29.590641 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" podUID="9753ddaa-3899-46ea-952f-5f03d76037c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:15:29.603689 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:29.603581 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:15:36.011371 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:15:36.011333 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:15:55.600979 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:55.591398 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" podUID="9753ddaa-3899-46ea-952f-5f03d76037c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:15:55.627317 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:55.627271 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" podUID="6d89a28a-7f87-4924-93c3-48ae7d11e56e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:15:56.597995 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:56.591906 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" podUID="d7037eab-3f5f-419b-94f6-784ff7331925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 02 14:15:56.624971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:56.598299 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" podUID="9753ddaa-3899-46ea-952f-5f03d76037c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:15:56.624971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:56.598680 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:15:56.655019 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:56.654097 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" podUID="6d89a28a-7f87-4924-93c3-48ae7d11e56e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded" Nov 02 14:15:56.655019 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:56.654671 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:15:56.764358 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:56.745998 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" podUID="0129df2e-9d21-45f7-9aa2-53777ce753b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:15:56.927537 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:56.865017 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" podUID="df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:15:56.932968 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:56.929715 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" podUID="4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:15:57.036815 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:57.025993 4866 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" podUID="0129df2e-9d21-45f7-9aa2-53777ce753b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:15:57.192034 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:57.190337 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" Nov 02 14:15:57.201438 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:15:57.201409 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:16:06.009979 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:16:06.009928 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:16:14.607972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:14.602279 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" podUID="d7037eab-3f5f-419b-94f6-784ff7331925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:14.686090 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:14.686041 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" podUID="df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:14.695057 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:14.695017 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" podUID="0129df2e-9d21-45f7-9aa2-53777ce753b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:14.892824 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:14.892718 4866 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" podUID="df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:26.186774 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.186735 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc"] Nov 02 14:16:26.187300 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.187286 4866 cpu_manager.go:414] "containerMap: removing container" podUID="9c90642e-dcd1-43af-b612-243f4af10796" containerName="revoke-keys" Nov 02 14:16:26.187300 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.187300 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c90642e-dcd1-43af-b612-243f4af10796" containerName="revoke-keys" Nov 02 14:16:26.187413 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.187404 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c90642e-dcd1-43af-b612-243f4af10796" containerName="revoke-keys" Nov 02 14:16:26.191291 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.191268 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.224414 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.224369 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc"] Nov 02 14:16:26.261330 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.261291 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.261496 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.261411 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.261496 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.261433 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqckh\" (UniqueName: \"kubernetes.io/projected/de984a72-1f11-42aa-a50a-1013f70af8f9-kube-api-access-mqckh\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.261496 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.261459 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.261627 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.261600 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.261667 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.261637 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.261729 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.261709 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.261811 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.261799 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.261864 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.261850 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de984a72-1f11-42aa-a50a-1013f70af8f9-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.291693 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.291666 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-866159-openshift-default-dockercfg-jpxc8\"" Nov 02 14:16:26.363165 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.363127 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.363347 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.363203 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.363347 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.363220 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqckh\" (UniqueName: \"kubernetes.io/projected/de984a72-1f11-42aa-a50a-1013f70af8f9-kube-api-access-mqckh\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.363347 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.363238 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.363347 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.363289 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.363347 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.363318 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.363613 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.363356 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.363613 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.363450 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.363613 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.363480 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de984a72-1f11-42aa-a50a-1013f70af8f9-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.363795 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.363770 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.363852 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.363840 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.363986 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.363960 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.364072 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.363985 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.364269 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.364249 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de984a72-1f11-42aa-a50a-1013f70af8f9-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.365531 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.365510 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.365787 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.365771 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.387112 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.387088 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.387267 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.387160 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqckh\" (UniqueName: \"kubernetes.io/projected/de984a72-1f11-42aa-a50a-1013f70af8f9-kube-api-access-mqckh\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.506751 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.506718 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:26.829171 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.829140 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc"] Nov 02 14:16:26.829813 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:16:26.829785 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde984a72_1f11_42aa_a50a_1013f70af8f9.slice/crio-60851121c6617d0c6922c43bd119710b18c83ce147e6acc04bde7ed462919c52 WatchSource:0}: Error finding container 60851121c6617d0c6922c43bd119710b18c83ce147e6acc04bde7ed462919c52: Status 404 returned error can't find the container with id 60851121c6617d0c6922c43bd119710b18c83ce147e6acc04bde7ed462919c52 Nov 02 14:16:26.832624 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.832588 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:16:26.832697 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.832671 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:16:26.832734 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:26.832713 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:16:27.563915 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:27.563877 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" event={"ID":"de984a72-1f11-42aa-a50a-1013f70af8f9","Type":"ContainerStarted","Data":"5a70d63d7b8eb7d874977324f633256203421e6014092560612136da9d298fb8"} Nov 02 14:16:27.564339 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:27.563938 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" event={"ID":"de984a72-1f11-42aa-a50a-1013f70af8f9","Type":"ContainerStarted","Data":"60851121c6617d0c6922c43bd119710b18c83ce147e6acc04bde7ed462919c52"} Nov 02 14:16:27.635017 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:27.634935 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" podStartSLOduration=1.6349160600000001 podStartE2EDuration="1.63491606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:16:27.621335114 +0000 UTC m=+1152.805182470" watchObservedRunningTime="2026-11-02 14:16:27.63491606 +0000 UTC m=+1152.818763398" Nov 02 14:16:28.523103 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:28.523065 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:28.523311 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:28.523143 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 14:16:28.523311 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:28.523273 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" podUID="de984a72-1f11-42aa-a50a-1013f70af8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 14:16:29.508970 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:29.508427 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 14:16:29.508970 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:29.508490 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" podUID="de984a72-1f11-42aa-a50a-1013f70af8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 14:16:30.512973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:30.509134 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 14:16:30.512973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:30.509207 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" podUID="de984a72-1f11-42aa-a50a-1013f70af8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 14:16:30.901135 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:30.901023 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" podUID="4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:31.217119 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:31.217064 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" podUID="9753ddaa-3899-46ea-952f-5f03d76037c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:31.507626 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:31.507527 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 14:16:31.507936 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:31.507887 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" podUID="de984a72-1f11-42aa-a50a-1013f70af8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 14:16:31.592100 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:31.592046 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" podUID="d7037eab-3f5f-419b-94f6-784ff7331925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:31.752332 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:31.752283 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" podUID="0129df2e-9d21-45f7-9aa2-53777ce753b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:31.766150 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:31.753763 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" podUID="df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:31.928804 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:31.925978 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" podUID="4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:31.928804 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:31.926698 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:16:32.220114 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:32.220038 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" podUID="9753ddaa-3899-46ea-952f-5f03d76037c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 02 14:16:32.222458 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:32.222433 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:16:32.358986 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:32.314747 4866 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:16:32.358986 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:32.314833 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" podUID="87976167-09be-407f-aa64-6e6a384739ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:32.577034 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:32.575970 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 14:16:32.577034 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:32.576034 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" podUID="de984a72-1f11-42aa-a50a-1013f70af8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 14:16:32.600972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:32.593039 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" podUID="d7037eab-3f5f-419b-94f6-784ff7331925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:32.600972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:32.594458 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:16:32.771998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:32.760247 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" podUID="df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:32.771998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:32.760624 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" podUID="0129df2e-9d21-45f7-9aa2-53777ce753b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Nov 02 14:16:32.771998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:32.761284 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:16:32.771998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:32.761342 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:16:32.797787 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:32.797671 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:16:32.829500 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:32.828993 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:16:32.875984 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:32.868499 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:16:32.898093 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:32.898025 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:16:32.937573 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:32.937548 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:16:33.517465 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:33.517421 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 14:16:33.523070 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:33.523017 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" podUID="de984a72-1f11-42aa-a50a-1013f70af8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 14:16:34.509378 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:34.509339 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 14:16:34.510022 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:34.509991 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" podUID="de984a72-1f11-42aa-a50a-1013f70af8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 14:16:35.512210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:35.512178 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:35.512621 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:35.512498 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:35.513328 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:16:35.513306 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:16:36.021107 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:16:36.021068 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:17:06.010282 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:17:06.010228 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:17:08.883065 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:08.879508 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh"] Nov 02 14:17:08.883065 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:08.879774 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" podUID="27f22147-746a-4848-bfe4-b31f89025f07" containerName="maas-api" containerID="cri-o://36460fdf7320caeea19e903cfb21aa905545ecc87a1f360ea1698c39c807e0ac" gracePeriod=30 Nov 02 14:17:09.659241 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:09.644395 4866 generic.go:356] "Generic (PLEG): container finished" podID="27f22147-746a-4848-bfe4-b31f89025f07" containerID="36460fdf7320caeea19e903cfb21aa905545ecc87a1f360ea1698c39c807e0ac" exitCode=0 Nov 02 14:17:09.659667 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:09.659022 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" event={"ID":"27f22147-746a-4848-bfe4-b31f89025f07","Type":"ContainerDied","Data":"36460fdf7320caeea19e903cfb21aa905545ecc87a1f360ea1698c39c807e0ac"} Nov 02 14:17:10.096280 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:10.096252 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" Nov 02 14:17:10.245710 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:10.245674 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f22147-746a-4848-bfe4-b31f89025f07-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f22147-746a-4848-bfe4-b31f89025f07-openshift-service-ca\") pod \"27f22147-746a-4848-bfe4-b31f89025f07\" (UID: \"27f22147-746a-4848-bfe4-b31f89025f07\") " Nov 02 14:17:10.245911 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:10.245782 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f22147-746a-4848-bfe4-b31f89025f07-kube-api-access-dqhmd\" (UniqueName: \"kubernetes.io/projected/27f22147-746a-4848-bfe4-b31f89025f07-kube-api-access-dqhmd\") pod \"27f22147-746a-4848-bfe4-b31f89025f07\" (UID: \"27f22147-746a-4848-bfe4-b31f89025f07\") " Nov 02 14:17:10.246000 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:10.245964 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27f22147-746a-4848-bfe4-b31f89025f07-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f22147-746a-4848-bfe4-b31f89025f07-metrics-certs\") pod \"27f22147-746a-4848-bfe4-b31f89025f07\" (UID: \"27f22147-746a-4848-bfe4-b31f89025f07\") " Nov 02 14:17:10.246076 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:10.246060 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27f22147-746a-4848-bfe4-b31f89025f07-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27f22147-746a-4848-bfe4-b31f89025f07-maas-api-tls\") pod \"27f22147-746a-4848-bfe4-b31f89025f07\" (UID: \"27f22147-746a-4848-bfe4-b31f89025f07\") " Nov 02 14:17:10.246148 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:10.246109 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f22147-746a-4848-bfe4-b31f89025f07-openshift-service-ca" pod "27f22147-746a-4848-bfe4-b31f89025f07" (UID: "27f22147-746a-4848-bfe4-b31f89025f07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:17:10.246562 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:10.246540 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f22147-746a-4848-bfe4-b31f89025f07-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:17:10.248003 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:10.247942 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f22147-746a-4848-bfe4-b31f89025f07-kube-api-access-dqhmd" pod "27f22147-746a-4848-bfe4-b31f89025f07" (UID: "27f22147-746a-4848-bfe4-b31f89025f07"). InnerVolumeSpecName "kube-api-access-dqhmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:17:10.248100 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:10.248075 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f22147-746a-4848-bfe4-b31f89025f07-maas-api-tls" pod "27f22147-746a-4848-bfe4-b31f89025f07" (UID: "27f22147-746a-4848-bfe4-b31f89025f07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:17:10.248142 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:10.248086 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f22147-746a-4848-bfe4-b31f89025f07-metrics-certs" pod "27f22147-746a-4848-bfe4-b31f89025f07" (UID: "27f22147-746a-4848-bfe4-b31f89025f07"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:17:10.347439 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:10.347402 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f22147-746a-4848-bfe4-b31f89025f07-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:17:10.347439 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:10.347431 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27f22147-746a-4848-bfe4-b31f89025f07-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:17:10.347439 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:10.347441 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqhmd\" (UniqueName: \"kubernetes.io/projected/27f22147-746a-4848-bfe4-b31f89025f07-kube-api-access-dqhmd\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:17:10.667307 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:10.667228 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" Nov 02 14:17:10.667307 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:10.667251 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh" event={"ID":"27f22147-746a-4848-bfe4-b31f89025f07","Type":"ContainerDied","Data":"78150070cbba23c4c0e4acab6798594c7b2c1bc5e2d090f1763a490eeeb4efea"} Nov 02 14:17:10.667547 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:10.667312 4866 scope.go:122] "RemoveContainer" containerID="36460fdf7320caeea19e903cfb21aa905545ecc87a1f360ea1698c39c807e0ac" Nov 02 14:17:10.720674 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:10.720632 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh"] Nov 02 14:17:10.732032 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:10.731997 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-de8cc7-5ff64c6ff9-w6twh"] Nov 02 14:17:10.952215 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:10.952172 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f22147-746a-4848-bfe4-b31f89025f07" path="/var/lib/kubelet/pods/27f22147-746a-4848-bfe4-b31f89025f07/volumes" Nov 02 14:17:11.343806 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.343728 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p"] Nov 02 14:17:11.344351 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.344334 4866 cpu_manager.go:414] "containerMap: removing container" podUID="27f22147-746a-4848-bfe4-b31f89025f07" containerName="maas-api" Nov 02 14:17:11.344397 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.344353 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f22147-746a-4848-bfe4-b31f89025f07" containerName="maas-api" Nov 02 14:17:11.344451 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.344442 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f22147-746a-4848-bfe4-b31f89025f07" containerName="maas-api" Nov 02 14:17:11.349029 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.349000 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.376961 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.376916 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p"] Nov 02 14:17:11.465336 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.465306 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88076e68-1465-4d45-9950-48b0b23fdda3-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.465521 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.465357 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.465521 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.465503 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88076e68-1465-4d45-9950-48b0b23fdda3-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.465598 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.465567 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.465678 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.465658 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qghkm\" (UniqueName: \"kubernetes.io/projected/88076e68-1465-4d45-9950-48b0b23fdda3-kube-api-access-qghkm\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.465762 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.465707 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.465762 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.465747 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88076e68-1465-4d45-9950-48b0b23fdda3-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.465826 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.465797 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.465856 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.465840 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.567544 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.567502 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88076e68-1465-4d45-9950-48b0b23fdda3-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.567724 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.567648 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.567724 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.567696 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qghkm\" (UniqueName: \"kubernetes.io/projected/88076e68-1465-4d45-9950-48b0b23fdda3-kube-api-access-qghkm\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.567803 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.567736 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.567837 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.567801 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88076e68-1465-4d45-9950-48b0b23fdda3-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.567879 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.567865 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.567918 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.567900 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.568016 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.567984 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88076e68-1465-4d45-9950-48b0b23fdda3-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.568085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.568043 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.568311 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.568278 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.568443 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.568347 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.568443 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.568362 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88076e68-1465-4d45-9950-48b0b23fdda3-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.568443 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.568372 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.568551 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.568504 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.570100 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.570078 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.570557 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.570539 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88076e68-1465-4d45-9950-48b0b23fdda3-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.591397 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.591373 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qghkm\" (UniqueName: \"kubernetes.io/projected/88076e68-1465-4d45-9950-48b0b23fdda3-kube-api-access-qghkm\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.592676 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.592651 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88076e68-1465-4d45-9950-48b0b23fdda3-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.668017 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.667905 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:11.988915 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.988889 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p"] Nov 02 14:17:11.991424 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:17:11.991381 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88076e68_1465_4d45_9950_48b0b23fdda3.slice/crio-a02ae1e3a83232d9528e67e4b75dfd5994edecb197798f51ad7a314385c2392c WatchSource:0}: Error finding container a02ae1e3a83232d9528e67e4b75dfd5994edecb197798f51ad7a314385c2392c: Status 404 returned error can't find the container with id a02ae1e3a83232d9528e67e4b75dfd5994edecb197798f51ad7a314385c2392c Nov 02 14:17:11.994794 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.994747 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:17:11.994873 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.994824 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:17:11.994873 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:11.994861 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:17:12.697701 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:12.697648 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" event={"ID":"88076e68-1465-4d45-9950-48b0b23fdda3","Type":"ContainerStarted","Data":"30aadfa937cbfc66afce4fbbab4860a76f26a3537ddcf944d45dd25b5c5b5cc3"} Nov 02 14:17:12.698148 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:12.697747 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" event={"ID":"88076e68-1465-4d45-9950-48b0b23fdda3","Type":"ContainerStarted","Data":"a02ae1e3a83232d9528e67e4b75dfd5994edecb197798f51ad7a314385c2392c"} Nov 02 14:17:12.746049 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:12.745993 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" podStartSLOduration=1.745975652 podStartE2EDuration="1.745975652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:12.744580306 +0000 UTC m=+1197.928427644" watchObservedRunningTime="2026-11-02 14:17:12.745975652 +0000 UTC m=+1197.929822988" Nov 02 14:17:13.668877 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:13.668843 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:13.670412 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:13.670388 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:17:13.670477 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:13.670450 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" podUID="88076e68-1465-4d45-9950-48b0b23fdda3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 14:17:14.668896 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:14.668863 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:17:14.669401 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:14.668965 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" podUID="88076e68-1465-4d45-9950-48b0b23fdda3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 14:17:15.063750 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:15.063724 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:17:15.065725 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:15.065703 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:17:15.668482 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:15.668440 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:17:15.668673 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:15.668513 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" podUID="88076e68-1465-4d45-9950-48b0b23fdda3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 14:17:16.668922 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:16.668882 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:17:16.669454 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:16.668942 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" podUID="88076e68-1465-4d45-9950-48b0b23fdda3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 14:17:17.692371 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:17.692333 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:17:17.693002 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:17.692967 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" podUID="88076e68-1465-4d45-9950-48b0b23fdda3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 14:17:18.668812 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:18.668777 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:17:18.669011 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:18.668842 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" podUID="88076e68-1465-4d45-9950-48b0b23fdda3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 14:17:19.672980 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:19.672932 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:19.673362 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:19.673243 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:19.673936 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:19.673914 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:17:31.853104 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:31.853053 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" podUID="d7037eab-3f5f-419b-94f6-784ff7331925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:17:31.858064 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:31.858015 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" podUID="9753ddaa-3899-46ea-952f-5f03d76037c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:17:31.871711 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:31.871672 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" podUID="4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 02 14:17:36.011306 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:17:36.011260 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:17:56.527996 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.527938 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr"] Nov 02 14:17:56.532266 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.532244 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.586304 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.586265 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr"] Nov 02 14:17:56.626682 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.626644 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.626874 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.626709 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.626874 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.626836 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.626989 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.626877 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.627050 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.627029 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.627101 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.627071 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc607e50-88ca-40d8-9a3b-1b1335f74098-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.627146 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.627132 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.627219 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.627203 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.627394 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.627374 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb7q6\" (UniqueName: \"kubernetes.io/projected/bc607e50-88ca-40d8-9a3b-1b1335f74098-kube-api-access-cb7q6\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.728760 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.728721 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.728937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.728779 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.728937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.728811 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.729075 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.728994 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.729140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.729086 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.729140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.729117 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc607e50-88ca-40d8-9a3b-1b1335f74098-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.729246 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.729162 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.729246 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.729178 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.729353 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.729244 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.729353 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.729276 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.729461 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.729362 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb7q6\" (UniqueName: \"kubernetes.io/projected/bc607e50-88ca-40d8-9a3b-1b1335f74098-kube-api-access-cb7q6\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.729461 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.729369 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.729543 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.729490 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.729752 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.729729 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc607e50-88ca-40d8-9a3b-1b1335f74098-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.731359 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.731331 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.731713 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.731691 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.753017 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.752992 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.756636 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.756611 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb7q6\" (UniqueName: \"kubernetes.io/projected/bc607e50-88ca-40d8-9a3b-1b1335f74098-kube-api-access-cb7q6\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:56.852520 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:56.852183 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:57.250935 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:57.250906 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr"] Nov 02 14:17:57.253343 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:17:57.253319 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc607e50_88ca_40d8_9a3b_1b1335f74098.slice/crio-b3078b404cf050b9a10e3e925011cb9456701487667c5c1fc22c2d7b0c8d7fec WatchSource:0}: Error finding container b3078b404cf050b9a10e3e925011cb9456701487667c5c1fc22c2d7b0c8d7fec: Status 404 returned error can't find the container with id b3078b404cf050b9a10e3e925011cb9456701487667c5c1fc22c2d7b0c8d7fec Nov 02 14:17:57.255721 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:57.255687 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:17:57.255806 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:57.255751 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:17:57.255806 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:57.255782 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:17:58.136624 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:58.136587 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" event={"ID":"bc607e50-88ca-40d8-9a3b-1b1335f74098","Type":"ContainerStarted","Data":"1ada60c8c2ff70b00a4ecb28c709dbc745c8b8e4fe562b55d3020b1f90dad329"} Nov 02 14:17:58.137059 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:58.136649 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" event={"ID":"bc607e50-88ca-40d8-9a3b-1b1335f74098","Type":"ContainerStarted","Data":"b3078b404cf050b9a10e3e925011cb9456701487667c5c1fc22c2d7b0c8d7fec"} Nov 02 14:17:58.229651 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:58.229590 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" podStartSLOduration=2.229570334 podStartE2EDuration="2.229570334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:58.216414195 +0000 UTC m=+1243.400261533" watchObservedRunningTime="2026-11-02 14:17:58.229570334 +0000 UTC m=+1243.413417672" Nov 02 14:17:58.861216 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:58.861175 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:17:58.864436 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:58.863201 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 14:17:58.864436 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:58.863258 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" podUID="bc607e50-88ca-40d8-9a3b-1b1335f74098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 14:17:59.853836 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:59.853793 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 14:17:59.854293 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:17:59.853858 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" podUID="bc607e50-88ca-40d8-9a3b-1b1335f74098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 14:18:00.853048 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:00.853018 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 14:18:00.853217 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:00.853066 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" podUID="bc607e50-88ca-40d8-9a3b-1b1335f74098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 14:18:01.853065 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:01.853030 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 14:18:01.853473 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:01.853094 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" podUID="bc607e50-88ca-40d8-9a3b-1b1335f74098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 14:18:02.853264 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:02.853224 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 14:18:02.853663 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:02.853278 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" podUID="bc607e50-88ca-40d8-9a3b-1b1335f74098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 14:18:03.853630 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:03.853591 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 14:18:03.854084 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:03.853657 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" podUID="bc607e50-88ca-40d8-9a3b-1b1335f74098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 14:18:04.857983 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:04.857942 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:18:04.858442 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:04.858430 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:18:04.859276 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:04.859257 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:18:06.012280 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:18:06.012242 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:18:36.045700 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:18:36.045653 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:18:47.367761 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.367725 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj"] Nov 02 14:18:47.372266 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.372244 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" Nov 02 14:18:47.400316 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.400284 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"agc-cred-636908050766\"" Nov 02 14:18:47.400689 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.400284 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"agc-cred-f650d0393004\"" Nov 02 14:18:47.400849 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.400311 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2bb4ef\"" Nov 02 14:18:47.400933 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.400369 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"routing-overlay\"" Nov 02 14:18:47.400933 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.400398 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"kube-root-ca.crt\"" Nov 02 14:18:47.400933 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.400449 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"openshift-service-ca.crt\"" Nov 02 14:18:47.400933 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.400467 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"agc-cred-2a51f04e5b66\"" Nov 02 14:18:47.400933 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.400591 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"payload-processing-external-model-e2e--c0104787-dockercfg-tx5xz\"" Nov 02 14:18:47.406554 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.406515 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj"] Nov 02 14:18:47.444187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.444144 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfm5g\" (UniqueName: \"kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-kube-api-access-qfm5g\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj\" (UID: \"c9843046-9668-46a7-87b3-f5d81a9987dd\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" Nov 02 14:18:47.444406 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.444276 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9843046-9668-46a7-87b3-f5d81a9987dd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj\" (UID: \"c9843046-9668-46a7-87b3-f5d81a9987dd\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" Nov 02 14:18:47.444406 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.444348 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9843046-9668-46a7-87b3-f5d81a9987dd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj\" (UID: \"c9843046-9668-46a7-87b3-f5d81a9987dd\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" Nov 02 14:18:47.444481 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.444427 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj\" (UID: \"c9843046-9668-46a7-87b3-f5d81a9987dd\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" Nov 02 14:18:47.545246 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.545215 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfm5g\" (UniqueName: \"kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-kube-api-access-qfm5g\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj\" (UID: \"c9843046-9668-46a7-87b3-f5d81a9987dd\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" Nov 02 14:18:47.545456 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.545299 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9843046-9668-46a7-87b3-f5d81a9987dd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj\" (UID: \"c9843046-9668-46a7-87b3-f5d81a9987dd\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" Nov 02 14:18:47.545456 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.545354 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9843046-9668-46a7-87b3-f5d81a9987dd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj\" (UID: \"c9843046-9668-46a7-87b3-f5d81a9987dd\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" Nov 02 14:18:47.545456 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.545402 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj\" (UID: \"c9843046-9668-46a7-87b3-f5d81a9987dd\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" Nov 02 14:18:47.546072 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.546047 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9843046-9668-46a7-87b3-f5d81a9987dd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj\" (UID: \"c9843046-9668-46a7-87b3-f5d81a9987dd\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" Nov 02 14:18:47.546299 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.546266 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9843046-9668-46a7-87b3-f5d81a9987dd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj\" (UID: \"c9843046-9668-46a7-87b3-f5d81a9987dd\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" Nov 02 14:18:47.548204 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.548179 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj\" (UID: \"c9843046-9668-46a7-87b3-f5d81a9987dd\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" Nov 02 14:18:47.568501 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.568474 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfm5g\" (UniqueName: \"kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-kube-api-access-qfm5g\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj\" (UID: \"c9843046-9668-46a7-87b3-f5d81a9987dd\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" Nov 02 14:18:47.687222 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.687141 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" Nov 02 14:18:47.990315 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:47.990284 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj"] Nov 02 14:18:47.994129 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:18:47.994050 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9843046_9668_46a7_87b3_f5d81a9987dd.slice/crio-623bb2d010623a9cc7a77d6f269d11fc7bbaab78e2b8297003b633fcd39ed686 WatchSource:0}: Error finding container 623bb2d010623a9cc7a77d6f269d11fc7bbaab78e2b8297003b633fcd39ed686: Status 404 returned error can't find the container with id 623bb2d010623a9cc7a77d6f269d11fc7bbaab78e2b8297003b633fcd39ed686 Nov 02 14:18:48.665745 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:48.665687 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" event={"ID":"c9843046-9668-46a7-87b3-f5d81a9987dd","Type":"ContainerStarted","Data":"8e92831860bc18ea7405bde42bca61643ae2a27ae00afa1e6f41dd3dcd31ea0e"} Nov 02 14:18:48.666250 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:48.665791 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" event={"ID":"c9843046-9668-46a7-87b3-f5d81a9987dd","Type":"ContainerStarted","Data":"623bb2d010623a9cc7a77d6f269d11fc7bbaab78e2b8297003b633fcd39ed686"} Nov 02 14:18:48.666250 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:48.665826 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" Nov 02 14:18:48.709861 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:48.709800 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" podStartSLOduration=1.709784551 podStartE2EDuration="1.709784551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:18:48.706964225 +0000 UTC m=+1293.890811573" watchObservedRunningTime="2026-11-02 14:18:48.709784551 +0000 UTC m=+1293.893631888" Nov 02 14:18:59.676201 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:18:59.676121 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" Nov 02 14:19:06.014856 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:19:06.014815 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:19:06.988857 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:06.988820 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn"] Nov 02 14:19:06.993050 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:06.993025 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" Nov 02 14:19:07.008305 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.008283 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 14:19:07.020393 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.020364 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 14:19:07.020393 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.020364 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 14:19:07.020800 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.020380 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-34c3e09c31c7\"" Nov 02 14:19:07.020800 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.020481 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-8bxpz\"" Nov 02 14:19:07.022831 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.022811 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 14:19:07.026788 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.026759 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn"] Nov 02 14:19:07.064801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.064766 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2tb7\" (UniqueName: \"kubernetes.io/projected/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-kube-api-access-f2tb7\") pod \"payload-processing-external-model-8569d75f4b-b6ftn\" (UID: \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\") " pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" Nov 02 14:19:07.065056 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.064915 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-routing-overlay-volume\") pod \"payload-processing-external-model-8569d75f4b-b6ftn\" (UID: \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\") " pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" Nov 02 14:19:07.065427 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.065399 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-provider-credentials\") pod \"payload-processing-external-model-8569d75f4b-b6ftn\" (UID: \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\") " pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" Nov 02 14:19:07.065537 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.065450 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-plugins-config-volume\") pod \"payload-processing-external-model-8569d75f4b-b6ftn\" (UID: \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\") " pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" Nov 02 14:19:07.181832 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.181793 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2tb7\" (UniqueName: \"kubernetes.io/projected/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-kube-api-access-f2tb7\") pod \"payload-processing-external-model-8569d75f4b-b6ftn\" (UID: \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\") " pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" Nov 02 14:19:07.183937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.183906 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-routing-overlay-volume\") pod \"payload-processing-external-model-8569d75f4b-b6ftn\" (UID: \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\") " pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" Nov 02 14:19:07.184345 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.184323 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-provider-credentials\") pod \"payload-processing-external-model-8569d75f4b-b6ftn\" (UID: \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\") " pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" Nov 02 14:19:07.200107 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.200047 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-plugins-config-volume\") pod \"payload-processing-external-model-8569d75f4b-b6ftn\" (UID: \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\") " pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" Nov 02 14:19:07.212400 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.201723 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-plugins-config-volume\") pod \"payload-processing-external-model-8569d75f4b-b6ftn\" (UID: \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\") " pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" Nov 02 14:19:07.217837 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.217583 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-routing-overlay-volume\") pod \"payload-processing-external-model-8569d75f4b-b6ftn\" (UID: \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\") " pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" Nov 02 14:19:07.240022 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.233218 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2tb7\" (UniqueName: \"kubernetes.io/projected/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-kube-api-access-f2tb7\") pod \"payload-processing-external-model-8569d75f4b-b6ftn\" (UID: \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\") " pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" Nov 02 14:19:07.241938 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.241908 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-provider-credentials\") pod \"payload-processing-external-model-8569d75f4b-b6ftn\" (UID: \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\") " pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" Nov 02 14:19:07.308345 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.308304 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" Nov 02 14:19:07.637484 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.637451 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn"] Nov 02 14:19:07.639311 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:19:07.639281 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2821f02_84e7_4a0b_8b7d_cf3f54e2a4e3.slice/crio-71564303709b56d135b953eca4dafa2e43b30cf28996103a7b6fa74be9f98782 WatchSource:0}: Error finding container 71564303709b56d135b953eca4dafa2e43b30cf28996103a7b6fa74be9f98782: Status 404 returned error can't find the container with id 71564303709b56d135b953eca4dafa2e43b30cf28996103a7b6fa74be9f98782 Nov 02 14:19:07.799119 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.799083 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" event={"ID":"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3","Type":"ContainerStarted","Data":"bbca98fdaf3cc661c489f339d9922b5e8cf3b613222541e27fe3a003d6987ee3"} Nov 02 14:19:07.799313 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.799141 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" event={"ID":"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3","Type":"ContainerStarted","Data":"71564303709b56d135b953eca4dafa2e43b30cf28996103a7b6fa74be9f98782"} Nov 02 14:19:07.799313 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.799169 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" Nov 02 14:19:07.844373 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:07.844322 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" podStartSLOduration=1.844307149 podStartE2EDuration="1.844307149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:19:07.839230577 +0000 UTC m=+1313.023077937" watchObservedRunningTime="2026-11-02 14:19:07.844307149 +0000 UTC m=+1313.028154509" Nov 02 14:19:18.810233 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:18.810202 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" Nov 02 14:19:36.010486 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:19:36.010441 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:19:57.211919 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:57.211889 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc"] Nov 02 14:19:57.216111 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:57.216089 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" Nov 02 14:19:57.246500 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:57.246470 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc"] Nov 02 14:19:57.304434 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:57.304404 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d153bea6-5bfd-4d35-aa31-6a2ccf697129-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc\" (UID: \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" Nov 02 14:19:57.304642 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:57.304448 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d153bea6-5bfd-4d35-aa31-6a2ccf697129-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc\" (UID: \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" Nov 02 14:19:57.304642 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:57.304470 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d153bea6-5bfd-4d35-aa31-6a2ccf697129-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc\" (UID: \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" Nov 02 14:19:57.304862 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:57.304640 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzlbf\" (UniqueName: \"kubernetes.io/projected/d153bea6-5bfd-4d35-aa31-6a2ccf697129-kube-api-access-fzlbf\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc\" (UID: \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" Nov 02 14:19:57.415073 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:57.415031 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d153bea6-5bfd-4d35-aa31-6a2ccf697129-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc\" (UID: \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" Nov 02 14:19:57.416022 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:57.415933 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d153bea6-5bfd-4d35-aa31-6a2ccf697129-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc\" (UID: \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" Nov 02 14:19:57.416294 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:57.416275 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d153bea6-5bfd-4d35-aa31-6a2ccf697129-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc\" (UID: \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" Nov 02 14:19:57.416505 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:57.416488 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzlbf\" (UniqueName: \"kubernetes.io/projected/d153bea6-5bfd-4d35-aa31-6a2ccf697129-kube-api-access-fzlbf\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc\" (UID: \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" Nov 02 14:19:57.429994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:57.424339 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d153bea6-5bfd-4d35-aa31-6a2ccf697129-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc\" (UID: \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" Nov 02 14:19:57.429994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:57.425467 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d153bea6-5bfd-4d35-aa31-6a2ccf697129-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc\" (UID: \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" Nov 02 14:19:57.429994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:57.426272 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d153bea6-5bfd-4d35-aa31-6a2ccf697129-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc\" (UID: \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" Nov 02 14:19:57.453250 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:57.453167 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzlbf\" (UniqueName: \"kubernetes.io/projected/d153bea6-5bfd-4d35-aa31-6a2ccf697129-kube-api-access-fzlbf\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc\" (UID: \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" Nov 02 14:19:57.530011 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:57.529915 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" Nov 02 14:19:57.850593 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:57.850560 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc"] Nov 02 14:19:57.852145 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:19:57.852119 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd153bea6_5bfd_4d35_aa31_6a2ccf697129.slice/crio-a1296ee64dd84d110388f1ee9b33a3e38da1ca3720211ba52051a9baacea11c0 WatchSource:0}: Error finding container a1296ee64dd84d110388f1ee9b33a3e38da1ca3720211ba52051a9baacea11c0: Status 404 returned error can't find the container with id a1296ee64dd84d110388f1ee9b33a3e38da1ca3720211ba52051a9baacea11c0 Nov 02 14:19:58.215444 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:58.215390 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" event={"ID":"d153bea6-5bfd-4d35-aa31-6a2ccf697129","Type":"ContainerStarted","Data":"345da9c8d53ad15b1850602748d1122ab3332accbab2d98ce3d3ddc5cb1960e7"} Nov 02 14:19:58.215896 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:58.215588 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" Nov 02 14:19:58.215896 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:58.215617 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" event={"ID":"d153bea6-5bfd-4d35-aa31-6a2ccf697129","Type":"ContainerStarted","Data":"a1296ee64dd84d110388f1ee9b33a3e38da1ca3720211ba52051a9baacea11c0"} Nov 02 14:19:58.259905 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:19:58.259853 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" podStartSLOduration=1.259839245 podStartE2EDuration="1.259839245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:19:58.256289625 +0000 UTC m=+1363.440136962" watchObservedRunningTime="2026-11-02 14:19:58.259839245 +0000 UTC m=+1363.443686582" Nov 02 14:20:02.087482 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:02.087441 4866 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2bb4ef/agc-cred-2a51f04e5b66: secret "agc-cred-2a51f04e5b66" not found Nov 02 14:20:02.087482 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:02.087473 4866 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2bb4ef/agc-cred-636908050766: secret "agc-cred-636908050766" not found Nov 02 14:20:02.087482 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:02.087489 4866 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj: [secret "agc-cred-2a51f04e5b66" not found, secret "agc-cred-636908050766" not found] Nov 02 14:20:02.088059 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:02.087551 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-provider-credentials podName:c9843046-9668-46a7-87b3-f5d81a9987dd nodeName:}" failed. No retries permitted until 2026-11-02 14:20:02.58753738 +0000 UTC m=+1367.771384708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" (UID: "c9843046-9668-46a7-87b3-f5d81a9987dd") : [secret "agc-cred-2a51f04e5b66" not found, secret "agc-cred-636908050766" not found] Nov 02 14:20:02.593218 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:02.593189 4866 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2bb4ef/agc-cred-2a51f04e5b66: secret "agc-cred-2a51f04e5b66" not found Nov 02 14:20:02.593218 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:02.593214 4866 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2bb4ef/agc-cred-636908050766: secret "agc-cred-636908050766" not found Nov 02 14:20:02.593218 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:02.593226 4866 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj: [secret "agc-cred-2a51f04e5b66" not found, secret "agc-cred-636908050766" not found] Nov 02 14:20:02.593556 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:02.593286 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-provider-credentials podName:c9843046-9668-46a7-87b3-f5d81a9987dd nodeName:}" failed. No retries permitted until 2026-11-02 14:20:03.593270678 +0000 UTC m=+1368.777118015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" (UID: "c9843046-9668-46a7-87b3-f5d81a9987dd") : [secret "agc-cred-2a51f04e5b66" not found, secret "agc-cred-636908050766" not found] Nov 02 14:20:03.607502 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:03.607467 4866 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2bb4ef/agc-cred-2a51f04e5b66: secret "agc-cred-2a51f04e5b66" not found Nov 02 14:20:03.607502 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:03.607497 4866 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2bb4ef/agc-cred-636908050766: secret "agc-cred-636908050766" not found Nov 02 14:20:03.607502 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:03.607511 4866 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj: [secret "agc-cred-2a51f04e5b66" not found, secret "agc-cred-636908050766" not found] Nov 02 14:20:03.607999 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:03.607569 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-provider-credentials podName:c9843046-9668-46a7-87b3-f5d81a9987dd nodeName:}" failed. No retries permitted until 2026-11-02 14:20:05.607553451 +0000 UTC m=+1370.791400782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" (UID: "c9843046-9668-46a7-87b3-f5d81a9987dd") : [secret "agc-cred-2a51f04e5b66" not found, secret "agc-cred-636908050766" not found] Nov 02 14:20:05.638827 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:05.638787 4866 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2bb4ef/agc-cred-2a51f04e5b66: secret "agc-cred-2a51f04e5b66" not found Nov 02 14:20:05.638827 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:05.638817 4866 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2bb4ef/agc-cred-636908050766: secret "agc-cred-636908050766" not found Nov 02 14:20:05.638827 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:05.638835 4866 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj: [secret "agc-cred-2a51f04e5b66" not found, secret "agc-cred-636908050766" not found] Nov 02 14:20:05.639360 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:05.638893 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-provider-credentials podName:c9843046-9668-46a7-87b3-f5d81a9987dd nodeName:}" failed. No retries permitted until 2026-11-02 14:20:09.638878361 +0000 UTC m=+1374.822725693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" (UID: "c9843046-9668-46a7-87b3-f5d81a9987dd") : [secret "agc-cred-2a51f04e5b66" not found, secret "agc-cred-636908050766" not found] Nov 02 14:20:06.010599 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:06.010563 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:20:09.036827 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:09.036788 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn"] Nov 02 14:20:09.047728 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:09.047693 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1b9b41b-664bn"] Nov 02 14:20:09.225831 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:09.225804 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" Nov 02 14:20:09.315482 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:09.315386 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj"] Nov 02 14:20:09.706063 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:09.706028 4866 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2bb4ef/agc-cred-2a51f04e5b66: secret "agc-cred-2a51f04e5b66" not found Nov 02 14:20:09.706063 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:09.706063 4866 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2bb4ef/agc-cred-636908050766: secret "agc-cred-636908050766" not found Nov 02 14:20:09.706296 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:09.706081 4866 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj: [secret "agc-cred-2a51f04e5b66" not found, secret "agc-cred-636908050766" not found] Nov 02 14:20:09.706296 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:09.706164 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-provider-credentials podName:c9843046-9668-46a7-87b3-f5d81a9987dd nodeName:}" failed. No retries permitted until 2026-11-02 14:20:17.706144103 +0000 UTC m=+1382.889991435 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" (UID: "c9843046-9668-46a7-87b3-f5d81a9987dd") : [secret "agc-cred-2a51f04e5b66" not found, secret "agc-cred-636908050766" not found] Nov 02 14:20:09.989699 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:09.989583 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc"] Nov 02 14:20:10.948519 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:10.948481 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c90642e-dcd1-43af-b612-243f4af10796" path="/var/lib/kubelet/pods/9c90642e-dcd1-43af-b612-243f4af10796/volumes" Nov 02 14:20:14.316412 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:14.316361 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" podUID="c9843046-9668-46a7-87b3-f5d81a9987dd" containerName="payload-processing" containerID="cri-o://8e92831860bc18ea7405bde42bca61643ae2a27ae00afa1e6f41dd3dcd31ea0e" gracePeriod=25 Nov 02 14:20:14.592129 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:14.592099 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" Nov 02 14:20:14.722013 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:14.721970 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9843046-9668-46a7-87b3-f5d81a9987dd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9843046-9668-46a7-87b3-f5d81a9987dd-routing-overlay-volume\") pod \"c9843046-9668-46a7-87b3-f5d81a9987dd\" (UID: \"c9843046-9668-46a7-87b3-f5d81a9987dd\") " Nov 02 14:20:14.722202 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:14.722037 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-provider-credentials\") pod \"c9843046-9668-46a7-87b3-f5d81a9987dd\" (UID: \"c9843046-9668-46a7-87b3-f5d81a9987dd\") " Nov 02 14:20:14.722202 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:14.722100 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-kube-api-access-qfm5g\" (UniqueName: \"kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-kube-api-access-qfm5g\") pod \"c9843046-9668-46a7-87b3-f5d81a9987dd\" (UID: \"c9843046-9668-46a7-87b3-f5d81a9987dd\") " Nov 02 14:20:14.722345 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:14.722303 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9843046-9668-46a7-87b3-f5d81a9987dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9843046-9668-46a7-87b3-f5d81a9987dd-plugins-config-volume\") pod \"c9843046-9668-46a7-87b3-f5d81a9987dd\" (UID: \"c9843046-9668-46a7-87b3-f5d81a9987dd\") " Nov 02 14:20:14.722401 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:14.722356 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9843046-9668-46a7-87b3-f5d81a9987dd-routing-overlay-volume" pod "c9843046-9668-46a7-87b3-f5d81a9987dd" (UID: "c9843046-9668-46a7-87b3-f5d81a9987dd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:20:14.722722 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:14.722695 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9843046-9668-46a7-87b3-f5d81a9987dd-plugins-config-volume" pod "c9843046-9668-46a7-87b3-f5d81a9987dd" (UID: "c9843046-9668-46a7-87b3-f5d81a9987dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:20:14.722995 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:14.722973 4866 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9843046-9668-46a7-87b3-f5d81a9987dd-routing-overlay-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.723081 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:14.723000 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9843046-9668-46a7-87b3-f5d81a9987dd-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.724306 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:14.724276 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-kube-api-access-qfm5g" pod "c9843046-9668-46a7-87b3-f5d81a9987dd" (UID: "c9843046-9668-46a7-87b3-f5d81a9987dd"). InnerVolumeSpecName "kube-api-access-qfm5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:14.724456 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:14.724431 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-provider-credentials" pod "c9843046-9668-46a7-87b3-f5d81a9987dd" (UID: "c9843046-9668-46a7-87b3-f5d81a9987dd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:14.826405 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:14.826311 4866 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-provider-credentials\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.826405 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:14.826348 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfm5g\" (UniqueName: \"kubernetes.io/projected/c9843046-9668-46a7-87b3-f5d81a9987dd-kube-api-access-qfm5g\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.990177 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:14.990121 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" podUID="d153bea6-5bfd-4d35-aa31-6a2ccf697129" containerName="payload-processing" containerID="cri-o://345da9c8d53ad15b1850602748d1122ab3332accbab2d98ce3d3ddc5cb1960e7" gracePeriod=25 Nov 02 14:20:15.249351 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.249324 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" Nov 02 14:20:15.333213 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.333179 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d153bea6-5bfd-4d35-aa31-6a2ccf697129-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d153bea6-5bfd-4d35-aa31-6a2ccf697129-routing-overlay-volume\") pod \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\" (UID: \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\") " Nov 02 14:20:15.333700 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.333221 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d153bea6-5bfd-4d35-aa31-6a2ccf697129-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d153bea6-5bfd-4d35-aa31-6a2ccf697129-plugins-config-volume\") pod \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\" (UID: \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\") " Nov 02 14:20:15.333700 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.333259 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d153bea6-5bfd-4d35-aa31-6a2ccf697129-kube-api-access-fzlbf\" (UniqueName: \"kubernetes.io/projected/d153bea6-5bfd-4d35-aa31-6a2ccf697129-kube-api-access-fzlbf\") pod \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\" (UID: \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\") " Nov 02 14:20:15.333700 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.333445 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d153bea6-5bfd-4d35-aa31-6a2ccf697129-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d153bea6-5bfd-4d35-aa31-6a2ccf697129-provider-credentials\") pod \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\" (UID: \"d153bea6-5bfd-4d35-aa31-6a2ccf697129\") " Nov 02 14:20:15.333700 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.333672 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d153bea6-5bfd-4d35-aa31-6a2ccf697129-routing-overlay-volume" pod "d153bea6-5bfd-4d35-aa31-6a2ccf697129" (UID: "d153bea6-5bfd-4d35-aa31-6a2ccf697129"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:20:15.334208 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.334181 4866 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d153bea6-5bfd-4d35-aa31-6a2ccf697129-routing-overlay-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:20:15.334490 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.334464 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d153bea6-5bfd-4d35-aa31-6a2ccf697129-plugins-config-volume" pod "d153bea6-5bfd-4d35-aa31-6a2ccf697129" (UID: "d153bea6-5bfd-4d35-aa31-6a2ccf697129"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:20:15.335845 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.335814 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d153bea6-5bfd-4d35-aa31-6a2ccf697129-kube-api-access-fzlbf" pod "d153bea6-5bfd-4d35-aa31-6a2ccf697129" (UID: "d153bea6-5bfd-4d35-aa31-6a2ccf697129"). InnerVolumeSpecName "kube-api-access-fzlbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:15.336152 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.336136 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d153bea6-5bfd-4d35-aa31-6a2ccf697129-provider-credentials" pod "d153bea6-5bfd-4d35-aa31-6a2ccf697129" (UID: "d153bea6-5bfd-4d35-aa31-6a2ccf697129"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:15.373197 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.373114 4866 generic.go:356] "Generic (PLEG): container finished" podID="d153bea6-5bfd-4d35-aa31-6a2ccf697129" containerID="345da9c8d53ad15b1850602748d1122ab3332accbab2d98ce3d3ddc5cb1960e7" exitCode=0 Nov 02 14:20:15.373197 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.373160 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" Nov 02 14:20:15.373413 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.373194 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" event={"ID":"d153bea6-5bfd-4d35-aa31-6a2ccf697129","Type":"ContainerDied","Data":"345da9c8d53ad15b1850602748d1122ab3332accbab2d98ce3d3ddc5cb1960e7"} Nov 02 14:20:15.373413 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.373251 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc" event={"ID":"d153bea6-5bfd-4d35-aa31-6a2ccf697129","Type":"ContainerDied","Data":"a1296ee64dd84d110388f1ee9b33a3e38da1ca3720211ba52051a9baacea11c0"} Nov 02 14:20:15.373413 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.373276 4866 scope.go:122] "RemoveContainer" containerID="345da9c8d53ad15b1850602748d1122ab3332accbab2d98ce3d3ddc5cb1960e7" Nov 02 14:20:15.374650 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.374628 4866 generic.go:356] "Generic (PLEG): container finished" podID="c9843046-9668-46a7-87b3-f5d81a9987dd" containerID="8e92831860bc18ea7405bde42bca61643ae2a27ae00afa1e6f41dd3dcd31ea0e" exitCode=0 Nov 02 14:20:15.374755 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.374679 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" Nov 02 14:20:15.374816 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.374759 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" event={"ID":"c9843046-9668-46a7-87b3-f5d81a9987dd","Type":"ContainerDied","Data":"8e92831860bc18ea7405bde42bca61643ae2a27ae00afa1e6f41dd3dcd31ea0e"} Nov 02 14:20:15.374816 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.374803 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj" event={"ID":"c9843046-9668-46a7-87b3-f5d81a9987dd","Type":"ContainerDied","Data":"623bb2d010623a9cc7a77d6f269d11fc7bbaab78e2b8297003b633fcd39ed686"} Nov 02 14:20:15.384978 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.384918 4866 scope.go:122] "RemoveContainer" containerID="345da9c8d53ad15b1850602748d1122ab3332accbab2d98ce3d3ddc5cb1960e7" Nov 02 14:20:15.385290 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:15.385271 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"345da9c8d53ad15b1850602748d1122ab3332accbab2d98ce3d3ddc5cb1960e7\": container with ID starting with 345da9c8d53ad15b1850602748d1122ab3332accbab2d98ce3d3ddc5cb1960e7 not found: ID does not exist" containerID="345da9c8d53ad15b1850602748d1122ab3332accbab2d98ce3d3ddc5cb1960e7" Nov 02 14:20:15.385351 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.385299 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"345da9c8d53ad15b1850602748d1122ab3332accbab2d98ce3d3ddc5cb1960e7"} err="failed to get container status \"345da9c8d53ad15b1850602748d1122ab3332accbab2d98ce3d3ddc5cb1960e7\": rpc error: code = NotFound desc = could not find container \"345da9c8d53ad15b1850602748d1122ab3332accbab2d98ce3d3ddc5cb1960e7\": container with ID starting with 345da9c8d53ad15b1850602748d1122ab3332accbab2d98ce3d3ddc5cb1960e7 not found: ID does not exist" Nov 02 14:20:15.385351 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.385317 4866 scope.go:122] "RemoveContainer" containerID="8e92831860bc18ea7405bde42bca61643ae2a27ae00afa1e6f41dd3dcd31ea0e" Nov 02 14:20:15.394094 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.394074 4866 scope.go:122] "RemoveContainer" containerID="8e92831860bc18ea7405bde42bca61643ae2a27ae00afa1e6f41dd3dcd31ea0e" Nov 02 14:20:15.394415 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:15.394396 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e92831860bc18ea7405bde42bca61643ae2a27ae00afa1e6f41dd3dcd31ea0e\": container with ID starting with 8e92831860bc18ea7405bde42bca61643ae2a27ae00afa1e6f41dd3dcd31ea0e not found: ID does not exist" containerID="8e92831860bc18ea7405bde42bca61643ae2a27ae00afa1e6f41dd3dcd31ea0e" Nov 02 14:20:15.394457 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.394424 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e92831860bc18ea7405bde42bca61643ae2a27ae00afa1e6f41dd3dcd31ea0e"} err="failed to get container status \"8e92831860bc18ea7405bde42bca61643ae2a27ae00afa1e6f41dd3dcd31ea0e\": rpc error: code = NotFound desc = could not find container \"8e92831860bc18ea7405bde42bca61643ae2a27ae00afa1e6f41dd3dcd31ea0e\": container with ID starting with 8e92831860bc18ea7405bde42bca61643ae2a27ae00afa1e6f41dd3dcd31ea0e not found: ID does not exist" Nov 02 14:20:15.430423 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.430389 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc"] Nov 02 14:20:15.433717 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.433689 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-85bdpb5vc"] Nov 02 14:20:15.435234 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.435208 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d153bea6-5bfd-4d35-aa31-6a2ccf697129-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:20:15.435336 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.435238 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzlbf\" (UniqueName: \"kubernetes.io/projected/d153bea6-5bfd-4d35-aa31-6a2ccf697129-kube-api-access-fzlbf\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:20:15.435336 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.435254 4866 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d153bea6-5bfd-4d35-aa31-6a2ccf697129-provider-credentials\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:20:15.477534 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.477492 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj"] Nov 02 14:20:15.477892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:15.477791 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f74njj"] Nov 02 14:20:16.947192 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:16.947161 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9843046-9668-46a7-87b3-f5d81a9987dd" path="/var/lib/kubelet/pods/c9843046-9668-46a7-87b3-f5d81a9987dd/volumes" Nov 02 14:20:16.947606 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:16.947592 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d153bea6-5bfd-4d35-aa31-6a2ccf697129" path="/var/lib/kubelet/pods/d153bea6-5bfd-4d35-aa31-6a2ccf697129/volumes" Nov 02 14:20:36.010657 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:20:36.010575 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:20:49.992307 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:49.992268 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz"] Nov 02 14:20:49.992856 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:49.992843 4866 cpu_manager.go:414] "containerMap: removing container" podUID="c9843046-9668-46a7-87b3-f5d81a9987dd" containerName="payload-processing" Nov 02 14:20:49.992899 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:49.992857 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9843046-9668-46a7-87b3-f5d81a9987dd" containerName="payload-processing" Nov 02 14:20:49.992899 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:49.992868 4866 cpu_manager.go:414] "containerMap: removing container" podUID="d153bea6-5bfd-4d35-aa31-6a2ccf697129" containerName="payload-processing" Nov 02 14:20:49.992899 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:49.992873 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d153bea6-5bfd-4d35-aa31-6a2ccf697129" containerName="payload-processing" Nov 02 14:20:49.993010 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:49.992975 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9843046-9668-46a7-87b3-f5d81a9987dd" containerName="payload-processing" Nov 02 14:20:49.993010 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:49.992996 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="d153bea6-5bfd-4d35-aa31-6a2ccf697129" containerName="payload-processing" Nov 02 14:20:49.996390 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:49.996374 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz" Nov 02 14:20:50.016724 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.016681 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz"] Nov 02 14:20:50.024599 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.024568 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c9fc7d82-dockercfg-4t8jx\"" Nov 02 14:20:50.024599 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.024569 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c9fc7d82\"" Nov 02 14:20:50.124854 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.124814 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98hh5\" (UniqueName: \"kubernetes.io/projected/3c8fe7a9-99fa-44e4-9afb-d09e5341bc51-kube-api-access-98hh5\") pod \"payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz\" (UID: \"3c8fe7a9-99fa-44e4-9afb-d09e5341bc51\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz" Nov 02 14:20:50.125470 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.125442 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8fe7a9-99fa-44e4-9afb-d09e5341bc51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz\" (UID: \"3c8fe7a9-99fa-44e4-9afb-d09e5341bc51\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz" Nov 02 14:20:50.126237 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.126218 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz"] Nov 02 14:20:50.141184 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.141152 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz"] Nov 02 14:20:50.153131 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.145194 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz" Nov 02 14:20:50.241203 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.241142 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d32030f-b0cf-4a74-9bc1-b6370469cd43-plugins-config-volume\") pod \"payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz\" (UID: \"1d32030f-b0cf-4a74-9bc1-b6370469cd43\") " pod="openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz" Nov 02 14:20:50.241399 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.241225 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8fe7a9-99fa-44e4-9afb-d09e5341bc51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz\" (UID: \"3c8fe7a9-99fa-44e4-9afb-d09e5341bc51\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz" Nov 02 14:20:50.241399 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.241337 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98hh5\" (UniqueName: \"kubernetes.io/projected/3c8fe7a9-99fa-44e4-9afb-d09e5341bc51-kube-api-access-98hh5\") pod \"payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz\" (UID: \"3c8fe7a9-99fa-44e4-9afb-d09e5341bc51\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz" Nov 02 14:20:50.241522 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.241421 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7pt2\" (UniqueName: \"kubernetes.io/projected/1d32030f-b0cf-4a74-9bc1-b6370469cd43-kube-api-access-c7pt2\") pod \"payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz\" (UID: \"1d32030f-b0cf-4a74-9bc1-b6370469cd43\") " pod="openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz" Nov 02 14:20:50.242707 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.242638 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8fe7a9-99fa-44e4-9afb-d09e5341bc51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz\" (UID: \"3c8fe7a9-99fa-44e4-9afb-d09e5341bc51\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz" Nov 02 14:20:50.269849 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.269802 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98hh5\" (UniqueName: \"kubernetes.io/projected/3c8fe7a9-99fa-44e4-9afb-d09e5341bc51-kube-api-access-98hh5\") pod \"payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz\" (UID: \"3c8fe7a9-99fa-44e4-9afb-d09e5341bc51\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz" Nov 02 14:20:50.310509 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.310465 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz" Nov 02 14:20:50.342721 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.342680 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7pt2\" (UniqueName: \"kubernetes.io/projected/1d32030f-b0cf-4a74-9bc1-b6370469cd43-kube-api-access-c7pt2\") pod \"payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz\" (UID: \"1d32030f-b0cf-4a74-9bc1-b6370469cd43\") " pod="openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz" Nov 02 14:20:50.342917 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.342838 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d32030f-b0cf-4a74-9bc1-b6370469cd43-plugins-config-volume\") pod \"payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz\" (UID: \"1d32030f-b0cf-4a74-9bc1-b6370469cd43\") " pod="openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz" Nov 02 14:20:50.343890 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.343861 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d32030f-b0cf-4a74-9bc1-b6370469cd43-plugins-config-volume\") pod \"payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz\" (UID: \"1d32030f-b0cf-4a74-9bc1-b6370469cd43\") " pod="openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz" Nov 02 14:20:50.367274 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.367249 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7pt2\" (UniqueName: \"kubernetes.io/projected/1d32030f-b0cf-4a74-9bc1-b6370469cd43-kube-api-access-c7pt2\") pod \"payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz\" (UID: \"1d32030f-b0cf-4a74-9bc1-b6370469cd43\") " pod="openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz" Nov 02 14:20:50.481419 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.481382 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz" Nov 02 14:20:50.619648 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.619613 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz"] Nov 02 14:20:50.626247 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:20:50.626214 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8fe7a9_99fa_44e4_9afb_d09e5341bc51.slice/crio-7f760fff311d41ad1e865f79ddd0479ef58a924370d5b5cec5d8e3486132e6bc WatchSource:0}: Error finding container 7f760fff311d41ad1e865f79ddd0479ef58a924370d5b5cec5d8e3486132e6bc: Status 404 returned error can't find the container with id 7f760fff311d41ad1e865f79ddd0479ef58a924370d5b5cec5d8e3486132e6bc Nov 02 14:20:50.785868 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:50.785203 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz"] Nov 02 14:20:51.613834 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:51.613793 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz" event={"ID":"3c8fe7a9-99fa-44e4-9afb-d09e5341bc51","Type":"ContainerStarted","Data":"0dc5425744c883922e812d06516e6105e4ce398c44fc536dcb5bcf7d7bc19226"} Nov 02 14:20:51.614334 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:51.613859 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz" event={"ID":"3c8fe7a9-99fa-44e4-9afb-d09e5341bc51","Type":"ContainerStarted","Data":"7f760fff311d41ad1e865f79ddd0479ef58a924370d5b5cec5d8e3486132e6bc"} Nov 02 14:20:51.614334 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:51.613889 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz" Nov 02 14:20:51.615511 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:51.615485 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz" event={"ID":"1d32030f-b0cf-4a74-9bc1-b6370469cd43","Type":"ContainerStarted","Data":"53cc5acbb7b42cbb5e58a3d01d39e9a8cbad236a336b74dd19c976d2f19d5ff6"} Nov 02 14:20:51.615625 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:51.615533 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz" Nov 02 14:20:51.615625 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:51.615553 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz" event={"ID":"1d32030f-b0cf-4a74-9bc1-b6370469cd43","Type":"ContainerStarted","Data":"794156f29e1722cf69cc09c772a338379f24487f1074f60e9e1ba23ff8f870a3"} Nov 02 14:20:51.656152 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:51.656098 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz" podStartSLOduration=2.6560838540000002 podStartE2EDuration="2.656083854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:51.654244698 +0000 UTC m=+1416.838092035" watchObservedRunningTime="2026-11-02 14:20:51.656083854 +0000 UTC m=+1416.839931191" Nov 02 14:20:51.697068 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:20:51.697002 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz" podStartSLOduration=1.696980966 podStartE2EDuration="1.696980966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:51.694157417 +0000 UTC m=+1416.878004755" watchObservedRunningTime="2026-11-02 14:20:51.696980966 +0000 UTC m=+1416.880828305" Nov 02 14:21:01.568597 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.568562 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm"] Nov 02 14:21:01.572899 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.572878 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" Nov 02 14:21:01.593232 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.593197 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm"] Nov 02 14:21:01.601522 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.601488 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"agc-cred-dec30ec8ff5a\"" Nov 02 14:21:01.601713 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.601553 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"openshift-service-ca.crt\"" Nov 02 14:21:01.601713 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.601553 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"payload-processing-external-model-e2e--3ffe20a0-dockercfg-kwvdn\"" Nov 02 14:21:01.601713 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.601584 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"agc-cred-047606d198bb\"" Nov 02 14:21:01.601713 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.601641 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"payload-processing-external-model-plugins-e2e-extmod-b-12b8da\"" Nov 02 14:21:01.601713 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.601652 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"kube-root-ca.crt\"" Nov 02 14:21:01.601713 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.601715 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"routing-overlay\"" Nov 02 14:21:01.711118 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.711079 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df59c7a3-9319-4916-a2c0-bb6b9c684086-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm\" (UID: \"df59c7a3-9319-4916-a2c0-bb6b9c684086\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" Nov 02 14:21:01.711289 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.711139 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df59c7a3-9319-4916-a2c0-bb6b9c684086-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm\" (UID: \"df59c7a3-9319-4916-a2c0-bb6b9c684086\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" Nov 02 14:21:01.711331 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.711299 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m2ff\" (UniqueName: \"kubernetes.io/projected/df59c7a3-9319-4916-a2c0-bb6b9c684086-kube-api-access-2m2ff\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm\" (UID: \"df59c7a3-9319-4916-a2c0-bb6b9c684086\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" Nov 02 14:21:01.711383 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.711364 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df59c7a3-9319-4916-a2c0-bb6b9c684086-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm\" (UID: \"df59c7a3-9319-4916-a2c0-bb6b9c684086\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" Nov 02 14:21:01.812516 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.812483 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df59c7a3-9319-4916-a2c0-bb6b9c684086-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm\" (UID: \"df59c7a3-9319-4916-a2c0-bb6b9c684086\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" Nov 02 14:21:01.812692 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.812535 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df59c7a3-9319-4916-a2c0-bb6b9c684086-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm\" (UID: \"df59c7a3-9319-4916-a2c0-bb6b9c684086\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" Nov 02 14:21:01.812692 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.812623 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m2ff\" (UniqueName: \"kubernetes.io/projected/df59c7a3-9319-4916-a2c0-bb6b9c684086-kube-api-access-2m2ff\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm\" (UID: \"df59c7a3-9319-4916-a2c0-bb6b9c684086\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" Nov 02 14:21:01.812805 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.812685 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df59c7a3-9319-4916-a2c0-bb6b9c684086-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm\" (UID: \"df59c7a3-9319-4916-a2c0-bb6b9c684086\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" Nov 02 14:21:01.813281 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.813257 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df59c7a3-9319-4916-a2c0-bb6b9c684086-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm\" (UID: \"df59c7a3-9319-4916-a2c0-bb6b9c684086\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" Nov 02 14:21:01.813474 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.813450 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df59c7a3-9319-4916-a2c0-bb6b9c684086-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm\" (UID: \"df59c7a3-9319-4916-a2c0-bb6b9c684086\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" Nov 02 14:21:01.815322 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.815293 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df59c7a3-9319-4916-a2c0-bb6b9c684086-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm\" (UID: \"df59c7a3-9319-4916-a2c0-bb6b9c684086\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" Nov 02 14:21:01.840118 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.840040 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m2ff\" (UniqueName: \"kubernetes.io/projected/df59c7a3-9319-4916-a2c0-bb6b9c684086-kube-api-access-2m2ff\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm\" (UID: \"df59c7a3-9319-4916-a2c0-bb6b9c684086\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" Nov 02 14:21:01.886889 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:01.886855 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" Nov 02 14:21:02.192181 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:02.192151 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm"] Nov 02 14:21:02.193336 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:21:02.193304 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf59c7a3_9319_4916_a2c0_bb6b9c684086.slice/crio-d69a5866962ba82bdfc39bfac27624b703f3dfc8c4c5a3620c484c3769a36d22 WatchSource:0}: Error finding container d69a5866962ba82bdfc39bfac27624b703f3dfc8c4c5a3620c484c3769a36d22: Status 404 returned error can't find the container with id d69a5866962ba82bdfc39bfac27624b703f3dfc8c4c5a3620c484c3769a36d22 Nov 02 14:21:02.624244 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:02.624164 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz" Nov 02 14:21:02.624244 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:02.624231 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz" Nov 02 14:21:02.730441 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:02.730402 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" event={"ID":"df59c7a3-9319-4916-a2c0-bb6b9c684086","Type":"ContainerStarted","Data":"a643ffc3f4d61a84061e8936fd8e6b9e62108a1facbb565e8b0a68f7580c0359"} Nov 02 14:21:02.730605 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:02.730466 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" event={"ID":"df59c7a3-9319-4916-a2c0-bb6b9c684086","Type":"ContainerStarted","Data":"d69a5866962ba82bdfc39bfac27624b703f3dfc8c4c5a3620c484c3769a36d22"} Nov 02 14:21:02.730605 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:02.730498 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" Nov 02 14:21:02.775250 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:02.775197 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" podStartSLOduration=1.7751807400000001 podStartE2EDuration="1.77518074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:02.771516075 +0000 UTC m=+1427.955363409" watchObservedRunningTime="2026-11-02 14:21:02.77518074 +0000 UTC m=+1427.959028077" Nov 02 14:21:06.013439 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:21:06.011494 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:21:07.685656 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:07.685624 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr"] Nov 02 14:21:07.689721 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:07.689703 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" Nov 02 14:21:07.705879 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:07.705847 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c9fc7d82-serving-cert\"" Nov 02 14:21:07.729649 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:07.729610 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr"] Nov 02 14:21:07.796968 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:07.796915 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpjmt\" (UniqueName: \"kubernetes.io/projected/91e373a6-85f3-486c-ade4-33ef0899c7d1-kube-api-access-vpjmt\") pod \"maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr\" (UID: \"91e373a6-85f3-486c-ade4-33ef0899c7d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" Nov 02 14:21:07.797158 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:07.797067 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e373a6-85f3-486c-ade4-33ef0899c7d1-openshift-service-ca\") pod \"maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr\" (UID: \"91e373a6-85f3-486c-ade4-33ef0899c7d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" Nov 02 14:21:07.797246 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:07.797225 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e373a6-85f3-486c-ade4-33ef0899c7d1-metrics-certs\") pod \"maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr\" (UID: \"91e373a6-85f3-486c-ade4-33ef0899c7d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" Nov 02 14:21:07.797315 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:07.797268 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91e373a6-85f3-486c-ade4-33ef0899c7d1-maas-api-tls\") pod \"maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr\" (UID: \"91e373a6-85f3-486c-ade4-33ef0899c7d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" Nov 02 14:21:07.902141 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:07.902101 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e373a6-85f3-486c-ade4-33ef0899c7d1-metrics-certs\") pod \"maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr\" (UID: \"91e373a6-85f3-486c-ade4-33ef0899c7d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" Nov 02 14:21:07.902493 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:07.902472 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91e373a6-85f3-486c-ade4-33ef0899c7d1-maas-api-tls\") pod \"maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr\" (UID: \"91e373a6-85f3-486c-ade4-33ef0899c7d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" Nov 02 14:21:07.902785 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:21:07.902689 4866 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-serving-cert: secret "maas-api-e2e-ait-c9fc7d82-serving-cert" not found Nov 02 14:21:07.902983 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:21:07.902969 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e373a6-85f3-486c-ade4-33ef0899c7d1-maas-api-tls podName:91e373a6-85f3-486c-ade4-33ef0899c7d1 nodeName:}" failed. No retries permitted until 2026-11-02 14:21:08.402923322 +0000 UTC m=+1433.586770651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/91e373a6-85f3-486c-ade4-33ef0899c7d1-maas-api-tls") pod "maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" (UID: "91e373a6-85f3-486c-ade4-33ef0899c7d1") : secret "maas-api-e2e-ait-c9fc7d82-serving-cert" not found Nov 02 14:21:07.903512 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:07.903482 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpjmt\" (UniqueName: \"kubernetes.io/projected/91e373a6-85f3-486c-ade4-33ef0899c7d1-kube-api-access-vpjmt\") pod \"maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr\" (UID: \"91e373a6-85f3-486c-ade4-33ef0899c7d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" Nov 02 14:21:07.905560 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:07.903967 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e373a6-85f3-486c-ade4-33ef0899c7d1-openshift-service-ca\") pod \"maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr\" (UID: \"91e373a6-85f3-486c-ade4-33ef0899c7d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" Nov 02 14:21:07.905560 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:07.904854 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e373a6-85f3-486c-ade4-33ef0899c7d1-openshift-service-ca\") pod \"maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr\" (UID: \"91e373a6-85f3-486c-ade4-33ef0899c7d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" Nov 02 14:21:07.907075 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:07.907053 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e373a6-85f3-486c-ade4-33ef0899c7d1-metrics-certs\") pod \"maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr\" (UID: \"91e373a6-85f3-486c-ade4-33ef0899c7d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" Nov 02 14:21:07.926878 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:07.926848 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpjmt\" (UniqueName: \"kubernetes.io/projected/91e373a6-85f3-486c-ade4-33ef0899c7d1-kube-api-access-vpjmt\") pod \"maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr\" (UID: \"91e373a6-85f3-486c-ade4-33ef0899c7d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" Nov 02 14:21:08.410784 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:08.410747 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91e373a6-85f3-486c-ade4-33ef0899c7d1-maas-api-tls\") pod \"maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr\" (UID: \"91e373a6-85f3-486c-ade4-33ef0899c7d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" Nov 02 14:21:08.413335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:08.413310 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91e373a6-85f3-486c-ade4-33ef0899c7d1-maas-api-tls\") pod \"maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr\" (UID: \"91e373a6-85f3-486c-ade4-33ef0899c7d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" Nov 02 14:21:08.605103 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:08.605063 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" Nov 02 14:21:08.934017 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:08.930080 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr"] Nov 02 14:21:08.957554 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:08.957523 4866 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:21:09.779318 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:09.779280 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" event={"ID":"91e373a6-85f3-486c-ade4-33ef0899c7d1","Type":"ContainerStarted","Data":"943ba3e8088235517ec2ea84f896aba33addb9913fe927480b568f5046e321ff"} Nov 02 14:21:09.779518 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:09.779349 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" event={"ID":"91e373a6-85f3-486c-ade4-33ef0899c7d1","Type":"ContainerStarted","Data":"919884d4022789fcbeb9f6f30a392032ba1eaec3ba43501c37a22102b7d15d1a"} Nov 02 14:21:09.779518 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:09.779504 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" Nov 02 14:21:09.822494 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:09.822415 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" podStartSLOduration=2.365336922 podStartE2EDuration="2.822396504s" totalImagesPullingTime="457.059582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:09.821095783 +0000 UTC m=+1435.004943121" watchObservedRunningTime="2026-11-02 14:21:09.822396504 +0000 UTC m=+1435.006243841" Nov 02 14:21:13.740968 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:13.740045 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" Nov 02 14:21:15.296871 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:15.296831 4866 scope.go:122] "RemoveContainer" containerID="4bb0d156829de41fcaff8d853ef2b92cb9602bb46292f59cf135bf9bbd9798e3" Nov 02 14:21:15.809154 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:15.809120 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" Nov 02 14:21:16.352973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:16.350473 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-259d8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:21:16.352973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:16.350537 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-259d8" podUID="cd410bed-b4ef-467a-95f9-9b520abfd44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:21:16.438969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:16.435864 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-9pqlk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:21:16.438969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:16.435969 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" podUID="e3459850-2b2b-4dc0-b1ab-5ffbc87263a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:21:36.011567 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:21:36.011528 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:21:45.321147 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:45.321115 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz"] Nov 02 14:21:45.377633 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:45.377596 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz"] Nov 02 14:21:50.322083 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:50.321999 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz" podUID="1d32030f-b0cf-4a74-9bc1-b6370469cd43" containerName="payload-processing" containerID="cri-o://53cc5acbb7b42cbb5e58a3d01d39e9a8cbad236a336b74dd19c976d2f19d5ff6" gracePeriod=25 Nov 02 14:21:50.378464 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:50.378416 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz" podUID="3c8fe7a9-99fa-44e4-9afb-d09e5341bc51" containerName="payload-pre-processing" containerID="cri-o://0dc5425744c883922e812d06516e6105e4ce398c44fc536dcb5bcf7d7bc19226" gracePeriod=25 Nov 02 14:21:50.692855 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:50.692825 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz" Nov 02 14:21:50.744841 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:50.744812 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d32030f-b0cf-4a74-9bc1-b6370469cd43-kube-api-access-c7pt2\" (UniqueName: \"kubernetes.io/projected/1d32030f-b0cf-4a74-9bc1-b6370469cd43-kube-api-access-c7pt2\") pod \"1d32030f-b0cf-4a74-9bc1-b6370469cd43\" (UID: \"1d32030f-b0cf-4a74-9bc1-b6370469cd43\") " Nov 02 14:21:50.744841 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:50.744847 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d32030f-b0cf-4a74-9bc1-b6370469cd43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d32030f-b0cf-4a74-9bc1-b6370469cd43-plugins-config-volume\") pod \"1d32030f-b0cf-4a74-9bc1-b6370469cd43\" (UID: \"1d32030f-b0cf-4a74-9bc1-b6370469cd43\") " Nov 02 14:21:50.745297 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:50.745274 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d32030f-b0cf-4a74-9bc1-b6370469cd43-plugins-config-volume" pod "1d32030f-b0cf-4a74-9bc1-b6370469cd43" (UID: "1d32030f-b0cf-4a74-9bc1-b6370469cd43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:21:50.745430 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:50.745414 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d32030f-b0cf-4a74-9bc1-b6370469cd43-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:21:50.747395 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:50.747367 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d32030f-b0cf-4a74-9bc1-b6370469cd43-kube-api-access-c7pt2" pod "1d32030f-b0cf-4a74-9bc1-b6370469cd43" (UID: "1d32030f-b0cf-4a74-9bc1-b6370469cd43"). InnerVolumeSpecName "kube-api-access-c7pt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:21:50.779444 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:50.779407 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz" Nov 02 14:21:50.846346 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:50.846316 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8fe7a9-99fa-44e4-9afb-d09e5341bc51-kube-api-access-98hh5\" (UniqueName: \"kubernetes.io/projected/3c8fe7a9-99fa-44e4-9afb-d09e5341bc51-kube-api-access-98hh5\") pod \"3c8fe7a9-99fa-44e4-9afb-d09e5341bc51\" (UID: \"3c8fe7a9-99fa-44e4-9afb-d09e5341bc51\") " Nov 02 14:21:50.846548 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:50.846373 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c8fe7a9-99fa-44e4-9afb-d09e5341bc51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8fe7a9-99fa-44e4-9afb-d09e5341bc51-plugins-config-volume\") pod \"3c8fe7a9-99fa-44e4-9afb-d09e5341bc51\" (UID: \"3c8fe7a9-99fa-44e4-9afb-d09e5341bc51\") " Nov 02 14:21:50.846792 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:50.846765 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8fe7a9-99fa-44e4-9afb-d09e5341bc51-plugins-config-volume" pod "3c8fe7a9-99fa-44e4-9afb-d09e5341bc51" (UID: "3c8fe7a9-99fa-44e4-9afb-d09e5341bc51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:21:50.847126 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:50.847105 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7pt2\" (UniqueName: \"kubernetes.io/projected/1d32030f-b0cf-4a74-9bc1-b6370469cd43-kube-api-access-c7pt2\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:21:50.847232 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:50.847128 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8fe7a9-99fa-44e4-9afb-d09e5341bc51-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:21:50.848669 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:50.848635 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8fe7a9-99fa-44e4-9afb-d09e5341bc51-kube-api-access-98hh5" pod "3c8fe7a9-99fa-44e4-9afb-d09e5341bc51" (UID: "3c8fe7a9-99fa-44e4-9afb-d09e5341bc51"). InnerVolumeSpecName "kube-api-access-98hh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:21:50.948074 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:50.948042 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98hh5\" (UniqueName: \"kubernetes.io/projected/3c8fe7a9-99fa-44e4-9afb-d09e5341bc51-kube-api-access-98hh5\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:21:51.246703 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:51.246611 4866 generic.go:356] "Generic (PLEG): container finished" podID="3c8fe7a9-99fa-44e4-9afb-d09e5341bc51" containerID="0dc5425744c883922e812d06516e6105e4ce398c44fc536dcb5bcf7d7bc19226" exitCode=0 Nov 02 14:21:51.246703 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:51.246660 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz" Nov 02 14:21:51.246966 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:51.246695 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz" event={"ID":"3c8fe7a9-99fa-44e4-9afb-d09e5341bc51","Type":"ContainerDied","Data":"0dc5425744c883922e812d06516e6105e4ce398c44fc536dcb5bcf7d7bc19226"} Nov 02 14:21:51.246966 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:51.246752 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz" event={"ID":"3c8fe7a9-99fa-44e4-9afb-d09e5341bc51","Type":"ContainerDied","Data":"7f760fff311d41ad1e865f79ddd0479ef58a924370d5b5cec5d8e3486132e6bc"} Nov 02 14:21:51.246966 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:51.246780 4866 scope.go:122] "RemoveContainer" containerID="0dc5425744c883922e812d06516e6105e4ce398c44fc536dcb5bcf7d7bc19226" Nov 02 14:21:51.248368 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:51.248343 4866 generic.go:356] "Generic (PLEG): container finished" podID="1d32030f-b0cf-4a74-9bc1-b6370469cd43" containerID="53cc5acbb7b42cbb5e58a3d01d39e9a8cbad236a336b74dd19c976d2f19d5ff6" exitCode=0 Nov 02 14:21:51.248471 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:51.248396 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz" Nov 02 14:21:51.248471 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:51.248408 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz" event={"ID":"1d32030f-b0cf-4a74-9bc1-b6370469cd43","Type":"ContainerDied","Data":"53cc5acbb7b42cbb5e58a3d01d39e9a8cbad236a336b74dd19c976d2f19d5ff6"} Nov 02 14:21:51.248471 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:51.248457 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz" event={"ID":"1d32030f-b0cf-4a74-9bc1-b6370469cd43","Type":"ContainerDied","Data":"794156f29e1722cf69cc09c772a338379f24487f1074f60e9e1ba23ff8f870a3"} Nov 02 14:21:51.256981 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:51.256957 4866 scope.go:122] "RemoveContainer" containerID="0dc5425744c883922e812d06516e6105e4ce398c44fc536dcb5bcf7d7bc19226" Nov 02 14:21:51.257346 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:21:51.257321 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc5425744c883922e812d06516e6105e4ce398c44fc536dcb5bcf7d7bc19226\": container with ID starting with 0dc5425744c883922e812d06516e6105e4ce398c44fc536dcb5bcf7d7bc19226 not found: ID does not exist" containerID="0dc5425744c883922e812d06516e6105e4ce398c44fc536dcb5bcf7d7bc19226" Nov 02 14:21:51.257417 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:51.257355 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc5425744c883922e812d06516e6105e4ce398c44fc536dcb5bcf7d7bc19226"} err="failed to get container status \"0dc5425744c883922e812d06516e6105e4ce398c44fc536dcb5bcf7d7bc19226\": rpc error: code = NotFound desc = could not find container \"0dc5425744c883922e812d06516e6105e4ce398c44fc536dcb5bcf7d7bc19226\": container with ID starting with 0dc5425744c883922e812d06516e6105e4ce398c44fc536dcb5bcf7d7bc19226 not found: ID does not exist" Nov 02 14:21:51.257417 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:51.257376 4866 scope.go:122] "RemoveContainer" containerID="53cc5acbb7b42cbb5e58a3d01d39e9a8cbad236a336b74dd19c976d2f19d5ff6" Nov 02 14:21:51.266660 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:51.266640 4866 scope.go:122] "RemoveContainer" containerID="53cc5acbb7b42cbb5e58a3d01d39e9a8cbad236a336b74dd19c976d2f19d5ff6" Nov 02 14:21:51.267014 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:21:51.266993 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53cc5acbb7b42cbb5e58a3d01d39e9a8cbad236a336b74dd19c976d2f19d5ff6\": container with ID starting with 53cc5acbb7b42cbb5e58a3d01d39e9a8cbad236a336b74dd19c976d2f19d5ff6 not found: ID does not exist" containerID="53cc5acbb7b42cbb5e58a3d01d39e9a8cbad236a336b74dd19c976d2f19d5ff6" Nov 02 14:21:51.267078 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:51.267023 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53cc5acbb7b42cbb5e58a3d01d39e9a8cbad236a336b74dd19c976d2f19d5ff6"} err="failed to get container status \"53cc5acbb7b42cbb5e58a3d01d39e9a8cbad236a336b74dd19c976d2f19d5ff6\": rpc error: code = NotFound desc = could not find container \"53cc5acbb7b42cbb5e58a3d01d39e9a8cbad236a336b74dd19c976d2f19d5ff6\": container with ID starting with 53cc5acbb7b42cbb5e58a3d01d39e9a8cbad236a336b74dd19c976d2f19d5ff6 not found: ID does not exist" Nov 02 14:21:51.299329 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:51.299296 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz"] Nov 02 14:21:51.307150 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:51.307117 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9fc7d82-74c8ff8767-k2mnz"] Nov 02 14:21:51.340489 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:51.340455 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz"] Nov 02 14:21:51.343834 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:51.343802 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9fc7d82-7b99bdf5dc-dqsrz"] Nov 02 14:21:52.947531 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:52.947489 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d32030f-b0cf-4a74-9bc1-b6370469cd43" path="/var/lib/kubelet/pods/1d32030f-b0cf-4a74-9bc1-b6370469cd43/volumes" Nov 02 14:21:52.948137 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:52.948117 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c8fe7a9-99fa-44e4-9afb-d09e5341bc51" path="/var/lib/kubelet/pods/3c8fe7a9-99fa-44e4-9afb-d09e5341bc51/volumes" Nov 02 14:21:53.818094 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:53.818054 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr"] Nov 02 14:21:53.818367 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:53.818318 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" podUID="91e373a6-85f3-486c-ade4-33ef0899c7d1" containerName="maas-api" containerID="cri-o://943ba3e8088235517ec2ea84f896aba33addb9913fe927480b568f5046e321ff" gracePeriod=30 Nov 02 14:21:54.169833 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.169799 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" Nov 02 14:21:54.234931 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.234880 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91e373a6-85f3-486c-ade4-33ef0899c7d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e373a6-85f3-486c-ade4-33ef0899c7d1-openshift-service-ca\") pod \"91e373a6-85f3-486c-ade4-33ef0899c7d1\" (UID: \"91e373a6-85f3-486c-ade4-33ef0899c7d1\") " Nov 02 14:21:54.234931 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.234959 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91e373a6-85f3-486c-ade4-33ef0899c7d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91e373a6-85f3-486c-ade4-33ef0899c7d1-maas-api-tls\") pod \"91e373a6-85f3-486c-ade4-33ef0899c7d1\" (UID: \"91e373a6-85f3-486c-ade4-33ef0899c7d1\") " Nov 02 14:21:54.235218 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.235120 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e373a6-85f3-486c-ade4-33ef0899c7d1-kube-api-access-vpjmt\" (UniqueName: \"kubernetes.io/projected/91e373a6-85f3-486c-ade4-33ef0899c7d1-kube-api-access-vpjmt\") pod \"91e373a6-85f3-486c-ade4-33ef0899c7d1\" (UID: \"91e373a6-85f3-486c-ade4-33ef0899c7d1\") " Nov 02 14:21:54.235218 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.235159 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91e373a6-85f3-486c-ade4-33ef0899c7d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e373a6-85f3-486c-ade4-33ef0899c7d1-metrics-certs\") pod \"91e373a6-85f3-486c-ade4-33ef0899c7d1\" (UID: \"91e373a6-85f3-486c-ade4-33ef0899c7d1\") " Nov 02 14:21:54.237811 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.237311 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e373a6-85f3-486c-ade4-33ef0899c7d1-openshift-service-ca" pod "91e373a6-85f3-486c-ade4-33ef0899c7d1" (UID: "91e373a6-85f3-486c-ade4-33ef0899c7d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:21:54.253983 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.249485 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91e373a6-85f3-486c-ade4-33ef0899c7d1-metrics-certs" pod "91e373a6-85f3-486c-ade4-33ef0899c7d1" (UID: "91e373a6-85f3-486c-ade4-33ef0899c7d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:21:54.275722 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.275676 4866 generic.go:356] "Generic (PLEG): container finished" podID="91e373a6-85f3-486c-ade4-33ef0899c7d1" containerID="943ba3e8088235517ec2ea84f896aba33addb9913fe927480b568f5046e321ff" exitCode=0 Nov 02 14:21:54.276990 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.275921 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" event={"ID":"91e373a6-85f3-486c-ade4-33ef0899c7d1","Type":"ContainerDied","Data":"943ba3e8088235517ec2ea84f896aba33addb9913fe927480b568f5046e321ff"} Nov 02 14:21:54.276990 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.276045 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" event={"ID":"91e373a6-85f3-486c-ade4-33ef0899c7d1","Type":"ContainerDied","Data":"919884d4022789fcbeb9f6f30a392032ba1eaec3ba43501c37a22102b7d15d1a"} Nov 02 14:21:54.276990 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.276086 4866 scope.go:122] "RemoveContainer" containerID="943ba3e8088235517ec2ea84f896aba33addb9913fe927480b568f5046e321ff" Nov 02 14:21:54.276990 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.276268 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr" Nov 02 14:21:54.277481 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.277450 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91e373a6-85f3-486c-ade4-33ef0899c7d1-maas-api-tls" pod "91e373a6-85f3-486c-ade4-33ef0899c7d1" (UID: "91e373a6-85f3-486c-ade4-33ef0899c7d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:21:54.301183 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.301089 4866 scope.go:122] "RemoveContainer" containerID="943ba3e8088235517ec2ea84f896aba33addb9913fe927480b568f5046e321ff" Nov 02 14:21:54.301824 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:21:54.301718 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"943ba3e8088235517ec2ea84f896aba33addb9913fe927480b568f5046e321ff\": container with ID starting with 943ba3e8088235517ec2ea84f896aba33addb9913fe927480b568f5046e321ff not found: ID does not exist" containerID="943ba3e8088235517ec2ea84f896aba33addb9913fe927480b568f5046e321ff" Nov 02 14:21:54.301824 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.301760 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"943ba3e8088235517ec2ea84f896aba33addb9913fe927480b568f5046e321ff"} err="failed to get container status \"943ba3e8088235517ec2ea84f896aba33addb9913fe927480b568f5046e321ff\": rpc error: code = NotFound desc = could not find container \"943ba3e8088235517ec2ea84f896aba33addb9913fe927480b568f5046e321ff\": container with ID starting with 943ba3e8088235517ec2ea84f896aba33addb9913fe927480b568f5046e321ff not found: ID does not exist" Nov 02 14:21:54.302873 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.302782 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e373a6-85f3-486c-ade4-33ef0899c7d1-kube-api-access-vpjmt" pod "91e373a6-85f3-486c-ade4-33ef0899c7d1" (UID: "91e373a6-85f3-486c-ade4-33ef0899c7d1"). InnerVolumeSpecName "kube-api-access-vpjmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:21:54.337601 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.337565 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e373a6-85f3-486c-ade4-33ef0899c7d1-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:21:54.337601 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.337597 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91e373a6-85f3-486c-ade4-33ef0899c7d1-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:21:54.337601 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.337612 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpjmt\" (UniqueName: \"kubernetes.io/projected/91e373a6-85f3-486c-ade4-33ef0899c7d1-kube-api-access-vpjmt\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:21:54.337850 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.337624 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e373a6-85f3-486c-ade4-33ef0899c7d1-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:21:54.624909 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.624866 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr"] Nov 02 14:21:54.629939 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.629903 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9fc7d82-867cfbfc76-nsgjr"] Nov 02 14:21:54.949983 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:54.949930 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e373a6-85f3-486c-ade4-33ef0899c7d1" path="/var/lib/kubelet/pods/91e373a6-85f3-486c-ade4-33ef0899c7d1/volumes" Nov 02 14:21:56.080257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:56.080219 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn"] Nov 02 14:21:59.524030 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.523513 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m"] Nov 02 14:21:59.524587 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.524508 4866 cpu_manager.go:414] "containerMap: removing container" podUID="1d32030f-b0cf-4a74-9bc1-b6370469cd43" containerName="payload-processing" Nov 02 14:21:59.524587 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.524533 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d32030f-b0cf-4a74-9bc1-b6370469cd43" containerName="payload-processing" Nov 02 14:21:59.524587 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.524549 4866 cpu_manager.go:414] "containerMap: removing container" podUID="3c8fe7a9-99fa-44e4-9afb-d09e5341bc51" containerName="payload-pre-processing" Nov 02 14:21:59.524587 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.524557 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c8fe7a9-99fa-44e4-9afb-d09e5341bc51" containerName="payload-pre-processing" Nov 02 14:21:59.524587 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.524567 4866 cpu_manager.go:414] "containerMap: removing container" podUID="91e373a6-85f3-486c-ade4-33ef0899c7d1" containerName="maas-api" Nov 02 14:21:59.524587 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.524575 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e373a6-85f3-486c-ade4-33ef0899c7d1" containerName="maas-api" Nov 02 14:21:59.524894 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.524750 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d32030f-b0cf-4a74-9bc1-b6370469cd43" containerName="payload-processing" Nov 02 14:21:59.524894 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.524764 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c8fe7a9-99fa-44e4-9afb-d09e5341bc51" containerName="payload-pre-processing" Nov 02 14:21:59.524894 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.524774 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e373a6-85f3-486c-ade4-33ef0899c7d1" containerName="maas-api" Nov 02 14:21:59.549002 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.548922 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" Nov 02 14:21:59.552474 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.552336 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m"] Nov 02 14:21:59.567221 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.567191 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"agc-cred-f36927adbb8c\"" Nov 02 14:21:59.663566 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.663530 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf657ad7-55b6-4568-8811-179057527b28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m\" (UID: \"cf657ad7-55b6-4568-8811-179057527b28\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" Nov 02 14:21:59.663781 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.663599 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4zh9\" (UniqueName: \"kubernetes.io/projected/cf657ad7-55b6-4568-8811-179057527b28-kube-api-access-w4zh9\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m\" (UID: \"cf657ad7-55b6-4568-8811-179057527b28\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" Nov 02 14:21:59.663889 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.663842 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf657ad7-55b6-4568-8811-179057527b28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m\" (UID: \"cf657ad7-55b6-4568-8811-179057527b28\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" Nov 02 14:21:59.664196 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.664158 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf657ad7-55b6-4568-8811-179057527b28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m\" (UID: \"cf657ad7-55b6-4568-8811-179057527b28\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" Nov 02 14:21:59.765345 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.765309 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf657ad7-55b6-4568-8811-179057527b28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m\" (UID: \"cf657ad7-55b6-4568-8811-179057527b28\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" Nov 02 14:21:59.765500 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.765381 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4zh9\" (UniqueName: \"kubernetes.io/projected/cf657ad7-55b6-4568-8811-179057527b28-kube-api-access-w4zh9\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m\" (UID: \"cf657ad7-55b6-4568-8811-179057527b28\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" Nov 02 14:21:59.765573 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.765533 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf657ad7-55b6-4568-8811-179057527b28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m\" (UID: \"cf657ad7-55b6-4568-8811-179057527b28\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" Nov 02 14:21:59.765613 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.765586 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf657ad7-55b6-4568-8811-179057527b28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m\" (UID: \"cf657ad7-55b6-4568-8811-179057527b28\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" Nov 02 14:21:59.766196 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.766169 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf657ad7-55b6-4568-8811-179057527b28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m\" (UID: \"cf657ad7-55b6-4568-8811-179057527b28\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" Nov 02 14:21:59.766266 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.766179 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf657ad7-55b6-4568-8811-179057527b28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m\" (UID: \"cf657ad7-55b6-4568-8811-179057527b28\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" Nov 02 14:21:59.768600 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.768578 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf657ad7-55b6-4568-8811-179057527b28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m\" (UID: \"cf657ad7-55b6-4568-8811-179057527b28\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" Nov 02 14:21:59.795019 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.794937 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4zh9\" (UniqueName: \"kubernetes.io/projected/cf657ad7-55b6-4568-8811-179057527b28-kube-api-access-w4zh9\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m\" (UID: \"cf657ad7-55b6-4568-8811-179057527b28\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" Nov 02 14:21:59.872728 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:21:59.872701 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" Nov 02 14:22:00.183183 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.183150 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m"] Nov 02 14:22:00.185082 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:22:00.185045 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf657ad7_55b6_4568_8811_179057527b28.slice/crio-d67c9df4f9eda93982168aa0a2e0e32a9d38d7a822cf48b35370a4b267eab20e WatchSource:0}: Error finding container d67c9df4f9eda93982168aa0a2e0e32a9d38d7a822cf48b35370a4b267eab20e: Status 404 returned error can't find the container with id d67c9df4f9eda93982168aa0a2e0e32a9d38d7a822cf48b35370a4b267eab20e Nov 02 14:22:00.357465 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.357418 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" event={"ID":"cf657ad7-55b6-4568-8811-179057527b28","Type":"ContainerStarted","Data":"46b67ae7abb0a0760e61b82646a537114564fb74c9bc1ffa094dabec112def6b"} Nov 02 14:22:00.357651 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.357496 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" event={"ID":"cf657ad7-55b6-4568-8811-179057527b28","Type":"ContainerStarted","Data":"d67c9df4f9eda93982168aa0a2e0e32a9d38d7a822cf48b35370a4b267eab20e"} Nov 02 14:22:00.357651 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.357614 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" Nov 02 14:22:00.397652 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.397612 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6"] Nov 02 14:22:00.402176 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.402152 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" Nov 02 14:22:00.408707 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.408632 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" podStartSLOduration=1.408613105 podStartE2EDuration="1.408613105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:00.406294327 +0000 UTC m=+1485.590141660" watchObservedRunningTime="2026-11-02 14:22:00.408613105 +0000 UTC m=+1485.592460443" Nov 02 14:22:00.419484 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.419460 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"agc-cred-b8b0c81894ae\"" Nov 02 14:22:00.419618 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.419530 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"agc-cred-598f75ee016d\"" Nov 02 14:22:00.419618 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.419572 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"agc-cred-d6506fd29d97\"" Nov 02 14:22:00.419618 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.419593 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"payload-processing-external-model-e2e--d8f5123f-dockercfg-qlpzh\"" Nov 02 14:22:00.419891 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.419859 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9ba664\"" Nov 02 14:22:00.420068 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.419880 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"routing-overlay\"" Nov 02 14:22:00.420068 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.419995 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"kube-root-ca.crt\"" Nov 02 14:22:00.420168 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.419995 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"openshift-service-ca.crt\"" Nov 02 14:22:00.423370 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.423339 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6"] Nov 02 14:22:00.576793 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.576755 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6\" (UID: \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" Nov 02 14:22:00.577169 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.576890 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q426d\" (UniqueName: \"kubernetes.io/projected/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-kube-api-access-q426d\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6\" (UID: \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" Nov 02 14:22:00.577169 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.576937 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6\" (UID: \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" Nov 02 14:22:00.577169 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.577033 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6\" (UID: \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" Nov 02 14:22:00.678097 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.678064 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6\" (UID: \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" Nov 02 14:22:00.678251 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.678153 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q426d\" (UniqueName: \"kubernetes.io/projected/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-kube-api-access-q426d\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6\" (UID: \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" Nov 02 14:22:00.678251 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.678179 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6\" (UID: \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" Nov 02 14:22:00.678251 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.678221 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6\" (UID: \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" Nov 02 14:22:00.678977 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.678929 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6\" (UID: \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" Nov 02 14:22:00.679053 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.679003 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6\" (UID: \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" Nov 02 14:22:00.680767 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.680750 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6\" (UID: \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" Nov 02 14:22:00.704659 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.704634 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q426d\" (UniqueName: \"kubernetes.io/projected/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-kube-api-access-q426d\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6\" (UID: \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" Nov 02 14:22:00.719626 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:00.719600 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" Nov 02 14:22:01.025188 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.025157 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6"] Nov 02 14:22:01.026350 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:22:01.026327 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747be1e1_d76f_4d33_bb59_1e8eb9af3c98.slice/crio-6e39831bae2ab13279d9e57d8b019a0fd75b3db3145db27e420e62e12e0f31b2 WatchSource:0}: Error finding container 6e39831bae2ab13279d9e57d8b019a0fd75b3db3145db27e420e62e12e0f31b2: Status 404 returned error can't find the container with id 6e39831bae2ab13279d9e57d8b019a0fd75b3db3145db27e420e62e12e0f31b2 Nov 02 14:22:01.081085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.081051 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" podUID="a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3" containerName="payload-processing" containerID="cri-o://bbca98fdaf3cc661c489f339d9922b5e8cf3b613222541e27fe3a003d6987ee3" gracePeriod=25 Nov 02 14:22:01.366325 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.366267 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" event={"ID":"747be1e1-d76f-4d33-bb59-1e8eb9af3c98","Type":"ContainerStarted","Data":"34628e78183f48c1a8e227acd7a9238ec6be6da99e9599d8876230a2532bb2c6"} Nov 02 14:22:01.366497 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.366343 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" event={"ID":"747be1e1-d76f-4d33-bb59-1e8eb9af3c98","Type":"ContainerStarted","Data":"6e39831bae2ab13279d9e57d8b019a0fd75b3db3145db27e420e62e12e0f31b2"} Nov 02 14:22:01.366497 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.366451 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" Nov 02 14:22:01.368025 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.367999 4866 generic.go:356] "Generic (PLEG): container finished" podID="a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3" containerID="bbca98fdaf3cc661c489f339d9922b5e8cf3b613222541e27fe3a003d6987ee3" exitCode=0 Nov 02 14:22:01.368139 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.368080 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" event={"ID":"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3","Type":"ContainerDied","Data":"bbca98fdaf3cc661c489f339d9922b5e8cf3b613222541e27fe3a003d6987ee3"} Nov 02 14:22:01.368210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.368138 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" event={"ID":"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3","Type":"ContainerDied","Data":"71564303709b56d135b953eca4dafa2e43b30cf28996103a7b6fa74be9f98782"} Nov 02 14:22:01.368210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.368162 4866 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71564303709b56d135b953eca4dafa2e43b30cf28996103a7b6fa74be9f98782" Nov 02 14:22:01.374746 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.374714 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" Nov 02 14:22:01.438901 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.438848 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" podStartSLOduration=1.438829261 podStartE2EDuration="1.438829261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:01.4368726 +0000 UTC m=+1486.620719937" watchObservedRunningTime="2026-11-02 14:22:01.438829261 +0000 UTC m=+1486.622676637" Nov 02 14:22:01.489230 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.489194 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-provider-credentials\") pod \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\" (UID: \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\") " Nov 02 14:22:01.489421 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.489403 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-routing-overlay-volume\") pod \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\" (UID: \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\") " Nov 02 14:22:01.489505 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.489436 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-plugins-config-volume\") pod \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\" (UID: \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\") " Nov 02 14:22:01.489505 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.489486 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-kube-api-access-f2tb7\" (UniqueName: \"kubernetes.io/projected/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-kube-api-access-f2tb7\") pod \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\" (UID: \"a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3\") " Nov 02 14:22:01.490005 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.489973 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-plugins-config-volume" pod "a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3" (UID: "a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:01.490348 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.490323 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:01.492678 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.492645 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-provider-credentials" pod "a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3" (UID: "a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:01.492926 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.492904 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-kube-api-access-f2tb7" pod "a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3" (UID: "a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3"). InnerVolumeSpecName "kube-api-access-f2tb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:01.493540 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.493514 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-routing-overlay-volume" pod "a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3" (UID: "a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:01.592333 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.592297 4866 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-routing-overlay-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:01.592333 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.592334 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2tb7\" (UniqueName: \"kubernetes.io/projected/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-kube-api-access-f2tb7\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:01.592880 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:01.592348 4866 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3-provider-credentials\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:02.373818 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:02.373790 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn" Nov 02 14:22:02.427093 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:02.427062 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn"] Nov 02 14:22:02.439339 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:02.439299 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-8569d75f4b-b6ftn"] Nov 02 14:22:02.947631 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:02.947598 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3" path="/var/lib/kubelet/pods/a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3/volumes" Nov 02 14:22:06.014817 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:22:06.014776 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:22:11.377413 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:11.377379 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" Nov 02 14:22:11.502413 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:11.502371 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm"] Nov 02 14:22:12.377655 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:12.377625 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" Nov 02 14:22:13.806573 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:13.806533 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm"] Nov 02 14:22:13.807554 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:13.807531 4866 cpu_manager.go:414] "containerMap: removing container" podUID="a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3" containerName="payload-processing" Nov 02 14:22:13.807613 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:13.807559 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3" containerName="payload-processing" Nov 02 14:22:13.807732 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:13.807718 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2821f02-84e7-4a0b-8b7d-cf3f54e2a4e3" containerName="payload-processing" Nov 02 14:22:13.811902 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:13.811879 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" Nov 02 14:22:13.831539 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:13.831506 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm"] Nov 02 14:22:13.844734 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:13.844697 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"agc-cred-fcbb8c928371\"" Nov 02 14:22:13.957552 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:13.957370 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4722986e-2218-4c2d-aac6-3b4a346e81ef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm\" (UID: \"4722986e-2218-4c2d-aac6-3b4a346e81ef\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" Nov 02 14:22:13.957552 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:13.957452 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4722986e-2218-4c2d-aac6-3b4a346e81ef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm\" (UID: \"4722986e-2218-4c2d-aac6-3b4a346e81ef\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" Nov 02 14:22:13.958546 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:13.958510 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-288cl\" (UniqueName: \"kubernetes.io/projected/4722986e-2218-4c2d-aac6-3b4a346e81ef-kube-api-access-288cl\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm\" (UID: \"4722986e-2218-4c2d-aac6-3b4a346e81ef\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" Nov 02 14:22:13.958975 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:13.958940 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4722986e-2218-4c2d-aac6-3b4a346e81ef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm\" (UID: \"4722986e-2218-4c2d-aac6-3b4a346e81ef\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" Nov 02 14:22:14.061026 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:14.060398 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4722986e-2218-4c2d-aac6-3b4a346e81ef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm\" (UID: \"4722986e-2218-4c2d-aac6-3b4a346e81ef\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" Nov 02 14:22:14.061026 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:14.060537 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4722986e-2218-4c2d-aac6-3b4a346e81ef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm\" (UID: \"4722986e-2218-4c2d-aac6-3b4a346e81ef\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" Nov 02 14:22:14.061026 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:14.060576 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4722986e-2218-4c2d-aac6-3b4a346e81ef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm\" (UID: \"4722986e-2218-4c2d-aac6-3b4a346e81ef\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" Nov 02 14:22:14.061026 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:14.060614 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-288cl\" (UniqueName: \"kubernetes.io/projected/4722986e-2218-4c2d-aac6-3b4a346e81ef-kube-api-access-288cl\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm\" (UID: \"4722986e-2218-4c2d-aac6-3b4a346e81ef\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" Nov 02 14:22:14.063002 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:14.061625 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4722986e-2218-4c2d-aac6-3b4a346e81ef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm\" (UID: \"4722986e-2218-4c2d-aac6-3b4a346e81ef\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" Nov 02 14:22:14.063002 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:14.062220 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4722986e-2218-4c2d-aac6-3b4a346e81ef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm\" (UID: \"4722986e-2218-4c2d-aac6-3b4a346e81ef\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" Nov 02 14:22:14.066996 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:14.064854 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4722986e-2218-4c2d-aac6-3b4a346e81ef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm\" (UID: \"4722986e-2218-4c2d-aac6-3b4a346e81ef\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" Nov 02 14:22:14.085148 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:14.085080 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-288cl\" (UniqueName: \"kubernetes.io/projected/4722986e-2218-4c2d-aac6-3b4a346e81ef-kube-api-access-288cl\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm\" (UID: \"4722986e-2218-4c2d-aac6-3b4a346e81ef\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" Nov 02 14:22:14.127248 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:14.127212 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" Nov 02 14:22:14.808971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:14.794428 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm"] Nov 02 14:22:15.322007 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:15.321307 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:22:15.322234 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:15.322013 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:22:15.787000 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:15.786936 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" event={"ID":"4722986e-2218-4c2d-aac6-3b4a346e81ef","Type":"ContainerStarted","Data":"bc6e13f8ee7effda3cf80ba92f54a9314c3454f4f7f3a1ee3d453608050f0650"} Nov 02 14:22:15.787367 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:15.787334 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" event={"ID":"4722986e-2218-4c2d-aac6-3b4a346e81ef","Type":"ContainerStarted","Data":"228be522a8e5e95373941887df99f7a972adc16c8391c105ec67e25d1ce9c9fa"} Nov 02 14:22:15.787713 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:15.787539 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" Nov 02 14:22:15.877033 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:15.860023 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" podStartSLOduration=2.859999374 podStartE2EDuration="2.859999374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:15.833359538 +0000 UTC m=+1501.017206876" watchObservedRunningTime="2026-11-02 14:22:15.859999374 +0000 UTC m=+1501.043846713" Nov 02 14:22:16.502964 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:16.502907 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" podUID="df59c7a3-9319-4916-a2c0-bb6b9c684086" containerName="payload-processing" containerID="cri-o://a643ffc3f4d61a84061e8936fd8e6b9e62108a1facbb565e8b0a68f7580c0359" gracePeriod=25 Nov 02 14:22:16.796429 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:16.796387 4866 generic.go:356] "Generic (PLEG): container finished" podID="df59c7a3-9319-4916-a2c0-bb6b9c684086" containerID="a643ffc3f4d61a84061e8936fd8e6b9e62108a1facbb565e8b0a68f7580c0359" exitCode=0 Nov 02 14:22:16.796587 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:16.796460 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" event={"ID":"df59c7a3-9319-4916-a2c0-bb6b9c684086","Type":"ContainerDied","Data":"a643ffc3f4d61a84061e8936fd8e6b9e62108a1facbb565e8b0a68f7580c0359"} Nov 02 14:22:16.796587 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:16.796514 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" event={"ID":"df59c7a3-9319-4916-a2c0-bb6b9c684086","Type":"ContainerDied","Data":"d69a5866962ba82bdfc39bfac27624b703f3dfc8c4c5a3620c484c3769a36d22"} Nov 02 14:22:16.796587 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:16.796530 4866 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d69a5866962ba82bdfc39bfac27624b703f3dfc8c4c5a3620c484c3769a36d22" Nov 02 14:22:16.810984 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:16.810939 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" Nov 02 14:22:16.923344 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:16.923304 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df59c7a3-9319-4916-a2c0-bb6b9c684086-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df59c7a3-9319-4916-a2c0-bb6b9c684086-routing-overlay-volume\") pod \"df59c7a3-9319-4916-a2c0-bb6b9c684086\" (UID: \"df59c7a3-9319-4916-a2c0-bb6b9c684086\") " Nov 02 14:22:16.923816 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:16.923382 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df59c7a3-9319-4916-a2c0-bb6b9c684086-provider-credentials\" (UniqueName: \"kubernetes.io/projected/df59c7a3-9319-4916-a2c0-bb6b9c684086-provider-credentials\") pod \"df59c7a3-9319-4916-a2c0-bb6b9c684086\" (UID: \"df59c7a3-9319-4916-a2c0-bb6b9c684086\") " Nov 02 14:22:16.923816 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:16.923508 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df59c7a3-9319-4916-a2c0-bb6b9c684086-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df59c7a3-9319-4916-a2c0-bb6b9c684086-plugins-config-volume\") pod \"df59c7a3-9319-4916-a2c0-bb6b9c684086\" (UID: \"df59c7a3-9319-4916-a2c0-bb6b9c684086\") " Nov 02 14:22:16.923816 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:16.923559 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df59c7a3-9319-4916-a2c0-bb6b9c684086-kube-api-access-2m2ff\" (UniqueName: \"kubernetes.io/projected/df59c7a3-9319-4916-a2c0-bb6b9c684086-kube-api-access-2m2ff\") pod \"df59c7a3-9319-4916-a2c0-bb6b9c684086\" (UID: \"df59c7a3-9319-4916-a2c0-bb6b9c684086\") " Nov 02 14:22:16.923816 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:16.923714 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df59c7a3-9319-4916-a2c0-bb6b9c684086-routing-overlay-volume" pod "df59c7a3-9319-4916-a2c0-bb6b9c684086" (UID: "df59c7a3-9319-4916-a2c0-bb6b9c684086"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:16.924192 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:16.924166 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df59c7a3-9319-4916-a2c0-bb6b9c684086-plugins-config-volume" pod "df59c7a3-9319-4916-a2c0-bb6b9c684086" (UID: "df59c7a3-9319-4916-a2c0-bb6b9c684086"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:16.924192 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:16.924183 4866 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df59c7a3-9319-4916-a2c0-bb6b9c684086-routing-overlay-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:16.926004 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:16.925973 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df59c7a3-9319-4916-a2c0-bb6b9c684086-kube-api-access-2m2ff" pod "df59c7a3-9319-4916-a2c0-bb6b9c684086" (UID: "df59c7a3-9319-4916-a2c0-bb6b9c684086"). InnerVolumeSpecName "kube-api-access-2m2ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:16.926224 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:16.926202 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df59c7a3-9319-4916-a2c0-bb6b9c684086-provider-credentials" pod "df59c7a3-9319-4916-a2c0-bb6b9c684086" (UID: "df59c7a3-9319-4916-a2c0-bb6b9c684086"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:17.026434 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:17.026355 4866 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df59c7a3-9319-4916-a2c0-bb6b9c684086-provider-credentials\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:17.026434 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:17.026387 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df59c7a3-9319-4916-a2c0-bb6b9c684086-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:17.026434 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:17.026403 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m2ff\" (UniqueName: \"kubernetes.io/projected/df59c7a3-9319-4916-a2c0-bb6b9c684086-kube-api-access-2m2ff\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:17.803516 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:17.803472 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm" Nov 02 14:22:17.866783 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:17.866745 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm"] Nov 02 14:22:17.880843 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:17.880797 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5plgsm"] Nov 02 14:22:18.951719 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:18.951670 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df59c7a3-9319-4916-a2c0-bb6b9c684086" path="/var/lib/kubelet/pods/df59c7a3-9319-4916-a2c0-bb6b9c684086/volumes" Nov 02 14:22:21.158679 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.158635 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps"] Nov 02 14:22:21.164007 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.160337 4866 cpu_manager.go:414] "containerMap: removing container" podUID="df59c7a3-9319-4916-a2c0-bb6b9c684086" containerName="payload-processing" Nov 02 14:22:21.164379 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.164347 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df59c7a3-9319-4916-a2c0-bb6b9c684086" containerName="payload-processing" Nov 02 14:22:21.165426 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.165403 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="df59c7a3-9319-4916-a2c0-bb6b9c684086" containerName="payload-processing" Nov 02 14:22:21.171367 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.171323 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" Nov 02 14:22:21.236409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.236365 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps"] Nov 02 14:22:21.237192 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.237162 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-87a1a06b-serving-cert\"" Nov 02 14:22:21.350171 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.350124 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn2l6\" (UniqueName: \"kubernetes.io/projected/34a67bee-8e3b-4029-b6e3-6ebae8070ead-kube-api-access-gn2l6\") pod \"maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps\" (UID: \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" Nov 02 14:22:21.350383 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.350185 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34a67bee-8e3b-4029-b6e3-6ebae8070ead-maas-api-tls\") pod \"maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps\" (UID: \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" Nov 02 14:22:21.350383 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.350316 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34a67bee-8e3b-4029-b6e3-6ebae8070ead-metrics-certs\") pod \"maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps\" (UID: \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" Nov 02 14:22:21.350513 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.350451 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a67bee-8e3b-4029-b6e3-6ebae8070ead-openshift-service-ca\") pod \"maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps\" (UID: \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" Nov 02 14:22:21.453240 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.453195 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a67bee-8e3b-4029-b6e3-6ebae8070ead-openshift-service-ca\") pod \"maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps\" (UID: \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" Nov 02 14:22:21.455601 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.455570 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn2l6\" (UniqueName: \"kubernetes.io/projected/34a67bee-8e3b-4029-b6e3-6ebae8070ead-kube-api-access-gn2l6\") pod \"maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps\" (UID: \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" Nov 02 14:22:21.455726 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.455621 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34a67bee-8e3b-4029-b6e3-6ebae8070ead-maas-api-tls\") pod \"maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps\" (UID: \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" Nov 02 14:22:21.455726 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.455652 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34a67bee-8e3b-4029-b6e3-6ebae8070ead-metrics-certs\") pod \"maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps\" (UID: \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" Nov 02 14:22:21.457563 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.454355 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a67bee-8e3b-4029-b6e3-6ebae8070ead-openshift-service-ca\") pod \"maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps\" (UID: \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" Nov 02 14:22:21.457844 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:22:21.457818 4866 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-serving-cert: secret "maas-api-e2e-migrate-87a1a06b-serving-cert" not found Nov 02 14:22:21.457959 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:22:21.457904 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34a67bee-8e3b-4029-b6e3-6ebae8070ead-maas-api-tls podName:34a67bee-8e3b-4029-b6e3-6ebae8070ead nodeName:}" failed. No retries permitted until 2026-11-02 14:22:21.957881769 +0000 UTC m=+1507.141729087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34a67bee-8e3b-4029-b6e3-6ebae8070ead-maas-api-tls") pod "maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" (UID: "34a67bee-8e3b-4029-b6e3-6ebae8070ead") : secret "maas-api-e2e-migrate-87a1a06b-serving-cert" not found Nov 02 14:22:21.460608 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.460581 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34a67bee-8e3b-4029-b6e3-6ebae8070ead-metrics-certs\") pod \"maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps\" (UID: \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" Nov 02 14:22:21.485891 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.485846 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn2l6\" (UniqueName: \"kubernetes.io/projected/34a67bee-8e3b-4029-b6e3-6ebae8070ead-kube-api-access-gn2l6\") pod \"maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps\" (UID: \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" Nov 02 14:22:21.962474 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.962442 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34a67bee-8e3b-4029-b6e3-6ebae8070ead-maas-api-tls\") pod \"maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps\" (UID: \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" Nov 02 14:22:21.964909 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:21.964882 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34a67bee-8e3b-4029-b6e3-6ebae8070ead-maas-api-tls\") pod \"maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps\" (UID: \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" Nov 02 14:22:22.180817 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:22.180780 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" Nov 02 14:22:22.496042 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:22.496006 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps"] Nov 02 14:22:22.497633 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:22:22.497606 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a67bee_8e3b_4029_b6e3_6ebae8070ead.slice/crio-777243ee7f6717263ab2b891ec8d19b0b19cbf530126b8bd6f1f5bd85c3a13b6 WatchSource:0}: Error finding container 777243ee7f6717263ab2b891ec8d19b0b19cbf530126b8bd6f1f5bd85c3a13b6: Status 404 returned error can't find the container with id 777243ee7f6717263ab2b891ec8d19b0b19cbf530126b8bd6f1f5bd85c3a13b6 Nov 02 14:22:22.837051 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:22.837014 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" event={"ID":"34a67bee-8e3b-4029-b6e3-6ebae8070ead","Type":"ContainerStarted","Data":"777243ee7f6717263ab2b891ec8d19b0b19cbf530126b8bd6f1f5bd85c3a13b6"} Nov 02 14:22:23.846569 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:23.846533 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" event={"ID":"34a67bee-8e3b-4029-b6e3-6ebae8070ead","Type":"ContainerStarted","Data":"23ab9581e067a2de8e8237c9cdf2328aebdf02af6dcca07a45b993b836d98ea3"} Nov 02 14:22:23.847050 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:23.846637 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" Nov 02 14:22:23.894005 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:23.893929 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" podStartSLOduration=2.5580435059999997 podStartE2EDuration="2.893915769s" totalImagesPullingTime="335.872263ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:23.889709461 +0000 UTC m=+1509.073556838" watchObservedRunningTime="2026-11-02 14:22:23.893915769 +0000 UTC m=+1509.077763106" Nov 02 14:22:25.917654 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:25.917609 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6"] Nov 02 14:22:26.810689 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:26.810653 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" Nov 02 14:22:26.913543 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:26.913496 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m"] Nov 02 14:22:29.860221 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:29.860188 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" Nov 02 14:22:30.918183 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:30.918121 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" podUID="747be1e1-d76f-4d33-bb59-1e8eb9af3c98" containerName="payload-processing" containerID="cri-o://34628e78183f48c1a8e227acd7a9238ec6be6da99e9599d8876230a2532bb2c6" gracePeriod=25 Nov 02 14:22:31.388094 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.388066 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" Nov 02 14:22:31.449539 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.449500 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-plugins-config-volume\") pod \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\" (UID: \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\") " Nov 02 14:22:31.449745 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.449682 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-kube-api-access-q426d\" (UniqueName: \"kubernetes.io/projected/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-kube-api-access-q426d\") pod \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\" (UID: \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\") " Nov 02 14:22:31.449821 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.449781 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-routing-overlay-volume\") pod \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\" (UID: \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\") " Nov 02 14:22:31.449821 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.449814 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-provider-credentials\" (UniqueName: \"kubernetes.io/projected/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-provider-credentials\") pod \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\" (UID: \"747be1e1-d76f-4d33-bb59-1e8eb9af3c98\") " Nov 02 14:22:31.450650 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.450137 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-plugins-config-volume" pod "747be1e1-d76f-4d33-bb59-1e8eb9af3c98" (UID: "747be1e1-d76f-4d33-bb59-1e8eb9af3c98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:31.450650 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.450371 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-routing-overlay-volume" pod "747be1e1-d76f-4d33-bb59-1e8eb9af3c98" (UID: "747be1e1-d76f-4d33-bb59-1e8eb9af3c98"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:31.450650 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.450602 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:31.450650 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.450623 4866 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-routing-overlay-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:31.452925 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.452859 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-kube-api-access-q426d" pod "747be1e1-d76f-4d33-bb59-1e8eb9af3c98" (UID: "747be1e1-d76f-4d33-bb59-1e8eb9af3c98"). InnerVolumeSpecName "kube-api-access-q426d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:31.453572 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.453544 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-provider-credentials" pod "747be1e1-d76f-4d33-bb59-1e8eb9af3c98" (UID: "747be1e1-d76f-4d33-bb59-1e8eb9af3c98"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:31.557152 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.557053 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q426d\" (UniqueName: \"kubernetes.io/projected/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-kube-api-access-q426d\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:31.557152 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.557091 4866 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/747be1e1-d76f-4d33-bb59-1e8eb9af3c98-provider-credentials\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:31.904925 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.904782 4866 generic.go:356] "Generic (PLEG): container finished" podID="747be1e1-d76f-4d33-bb59-1e8eb9af3c98" containerID="34628e78183f48c1a8e227acd7a9238ec6be6da99e9599d8876230a2532bb2c6" exitCode=0 Nov 02 14:22:31.904925 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.904842 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" Nov 02 14:22:31.904925 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.904873 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" event={"ID":"747be1e1-d76f-4d33-bb59-1e8eb9af3c98","Type":"ContainerDied","Data":"34628e78183f48c1a8e227acd7a9238ec6be6da99e9599d8876230a2532bb2c6"} Nov 02 14:22:31.904925 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.904929 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6" event={"ID":"747be1e1-d76f-4d33-bb59-1e8eb9af3c98","Type":"ContainerDied","Data":"6e39831bae2ab13279d9e57d8b019a0fd75b3db3145db27e420e62e12e0f31b2"} Nov 02 14:22:31.923994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.904990 4866 scope.go:122] "RemoveContainer" containerID="34628e78183f48c1a8e227acd7a9238ec6be6da99e9599d8876230a2532bb2c6" Nov 02 14:22:31.923994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.914408 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" podUID="cf657ad7-55b6-4568-8811-179057527b28" containerName="payload-processing" containerID="cri-o://46b67ae7abb0a0760e61b82646a537114564fb74c9bc1ffa094dabec112def6b" gracePeriod=25 Nov 02 14:22:31.923994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.920034 4866 scope.go:122] "RemoveContainer" containerID="34628e78183f48c1a8e227acd7a9238ec6be6da99e9599d8876230a2532bb2c6" Nov 02 14:22:31.923994 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:22:31.920549 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34628e78183f48c1a8e227acd7a9238ec6be6da99e9599d8876230a2532bb2c6\": container with ID starting with 34628e78183f48c1a8e227acd7a9238ec6be6da99e9599d8876230a2532bb2c6 not found: ID does not exist" containerID="34628e78183f48c1a8e227acd7a9238ec6be6da99e9599d8876230a2532bb2c6" Nov 02 14:22:31.923994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.920578 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34628e78183f48c1a8e227acd7a9238ec6be6da99e9599d8876230a2532bb2c6"} err="failed to get container status \"34628e78183f48c1a8e227acd7a9238ec6be6da99e9599d8876230a2532bb2c6\": rpc error: code = NotFound desc = could not find container \"34628e78183f48c1a8e227acd7a9238ec6be6da99e9599d8876230a2532bb2c6\": container with ID starting with 34628e78183f48c1a8e227acd7a9238ec6be6da99e9599d8876230a2532bb2c6 not found: ID does not exist" Nov 02 14:22:31.960119 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.960088 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6"] Nov 02 14:22:31.973257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:31.973222 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5d7fkk6"] Nov 02 14:22:32.206686 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:32.206660 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" Nov 02 14:22:32.270643 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:32.270595 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf657ad7-55b6-4568-8811-179057527b28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf657ad7-55b6-4568-8811-179057527b28-plugins-config-volume\") pod \"cf657ad7-55b6-4568-8811-179057527b28\" (UID: \"cf657ad7-55b6-4568-8811-179057527b28\") " Nov 02 14:22:32.270824 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:32.270651 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf657ad7-55b6-4568-8811-179057527b28-kube-api-access-w4zh9\" (UniqueName: \"kubernetes.io/projected/cf657ad7-55b6-4568-8811-179057527b28-kube-api-access-w4zh9\") pod \"cf657ad7-55b6-4568-8811-179057527b28\" (UID: \"cf657ad7-55b6-4568-8811-179057527b28\") " Nov 02 14:22:32.270824 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:32.270712 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf657ad7-55b6-4568-8811-179057527b28-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf657ad7-55b6-4568-8811-179057527b28-routing-overlay-volume\") pod \"cf657ad7-55b6-4568-8811-179057527b28\" (UID: \"cf657ad7-55b6-4568-8811-179057527b28\") " Nov 02 14:22:32.270961 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:32.270880 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf657ad7-55b6-4568-8811-179057527b28-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf657ad7-55b6-4568-8811-179057527b28-provider-credentials\") pod \"cf657ad7-55b6-4568-8811-179057527b28\" (UID: \"cf657ad7-55b6-4568-8811-179057527b28\") " Nov 02 14:22:32.271186 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:32.271157 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf657ad7-55b6-4568-8811-179057527b28-plugins-config-volume" pod "cf657ad7-55b6-4568-8811-179057527b28" (UID: "cf657ad7-55b6-4568-8811-179057527b28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:32.271251 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:32.271190 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf657ad7-55b6-4568-8811-179057527b28-routing-overlay-volume" pod "cf657ad7-55b6-4568-8811-179057527b28" (UID: "cf657ad7-55b6-4568-8811-179057527b28"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:32.271673 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:32.271645 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf657ad7-55b6-4568-8811-179057527b28-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:32.271770 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:32.271675 4866 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf657ad7-55b6-4568-8811-179057527b28-routing-overlay-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:32.273303 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:32.273269 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf657ad7-55b6-4568-8811-179057527b28-kube-api-access-w4zh9" pod "cf657ad7-55b6-4568-8811-179057527b28" (UID: "cf657ad7-55b6-4568-8811-179057527b28"). InnerVolumeSpecName "kube-api-access-w4zh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:32.273525 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:32.273505 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf657ad7-55b6-4568-8811-179057527b28-provider-credentials" pod "cf657ad7-55b6-4568-8811-179057527b28" (UID: "cf657ad7-55b6-4568-8811-179057527b28"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:32.373127 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:32.373092 4866 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf657ad7-55b6-4568-8811-179057527b28-provider-credentials\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:32.373127 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:32.373123 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4zh9\" (UniqueName: \"kubernetes.io/projected/cf657ad7-55b6-4568-8811-179057527b28-kube-api-access-w4zh9\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:33.069352 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:33.069309 4866 generic.go:356] "Generic (PLEG): container finished" podID="cf657ad7-55b6-4568-8811-179057527b28" containerID="46b67ae7abb0a0760e61b82646a537114564fb74c9bc1ffa094dabec112def6b" exitCode=0 Nov 02 14:22:33.069873 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:33.069667 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="747be1e1-d76f-4d33-bb59-1e8eb9af3c98" path="/var/lib/kubelet/pods/747be1e1-d76f-4d33-bb59-1e8eb9af3c98/volumes" Nov 02 14:22:33.069873 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:33.069727 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" Nov 02 14:22:33.070415 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:33.070386 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" event={"ID":"cf657ad7-55b6-4568-8811-179057527b28","Type":"ContainerDied","Data":"46b67ae7abb0a0760e61b82646a537114564fb74c9bc1ffa094dabec112def6b"} Nov 02 14:22:33.070539 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:33.070518 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m" event={"ID":"cf657ad7-55b6-4568-8811-179057527b28","Type":"ContainerDied","Data":"d67c9df4f9eda93982168aa0a2e0e32a9d38d7a822cf48b35370a4b267eab20e"} Nov 02 14:22:33.070804 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:33.070782 4866 scope.go:122] "RemoveContainer" containerID="46b67ae7abb0a0760e61b82646a537114564fb74c9bc1ffa094dabec112def6b" Nov 02 14:22:33.088993 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:33.088969 4866 scope.go:122] "RemoveContainer" containerID="46b67ae7abb0a0760e61b82646a537114564fb74c9bc1ffa094dabec112def6b" Nov 02 14:22:33.099157 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:22:33.099117 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46b67ae7abb0a0760e61b82646a537114564fb74c9bc1ffa094dabec112def6b\": container with ID starting with 46b67ae7abb0a0760e61b82646a537114564fb74c9bc1ffa094dabec112def6b not found: ID does not exist" containerID="46b67ae7abb0a0760e61b82646a537114564fb74c9bc1ffa094dabec112def6b" Nov 02 14:22:33.099326 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:33.099171 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b67ae7abb0a0760e61b82646a537114564fb74c9bc1ffa094dabec112def6b"} err="failed to get container status \"46b67ae7abb0a0760e61b82646a537114564fb74c9bc1ffa094dabec112def6b\": rpc error: code = NotFound desc = could not find container \"46b67ae7abb0a0760e61b82646a537114564fb74c9bc1ffa094dabec112def6b\": container with ID starting with 46b67ae7abb0a0760e61b82646a537114564fb74c9bc1ffa094dabec112def6b not found: ID does not exist" Nov 02 14:22:33.119621 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:22:33.119583 4866 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf657ad7_55b6_4568_8811_179057527b28.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf657ad7_55b6_4568_8811_179057527b28.slice/crio-d67c9df4f9eda93982168aa0a2e0e32a9d38d7a822cf48b35370a4b267eab20e\": RecentStats: unable to find data in memory cache]" Nov 02 14:22:33.140264 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:33.140210 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m"] Nov 02 14:22:33.149841 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:33.149801 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688fgqq7m"] Nov 02 14:22:34.968324 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:34.968187 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf657ad7-55b6-4568-8811-179057527b28" path="/var/lib/kubelet/pods/cf657ad7-55b6-4568-8811-179057527b28/volumes" Nov 02 14:22:36.086186 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:22:36.086147 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:22:39.855874 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:39.855836 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg"] Nov 02 14:22:39.856326 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:39.856098 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" podUID="8ac034d4-7e4f-468d-8a4d-b712f4d81bae" containerName="istio-proxy" containerID="cri-o://fe245455cbf9c36166526f4b712cb01d1d7bc73d40e473db06c2f9322aae81fa" gracePeriod=30 Nov 02 14:22:39.956739 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:39.956705 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8"] Nov 02 14:22:39.956932 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:39.956909 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" podUID="65cbb15a-d104-4b88-9147-290d212c6842" containerName="istio-proxy" containerID="cri-o://d533a9420cde800e9a72a49b5c82a9a2b1b9f41195bba232db215b97b2055188" gracePeriod=30 Nov 02 14:22:45.153172 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.153135 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:22:45.193671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.192010 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-data\") pod \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " Nov 02 14:22:45.193671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.192109 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-workload-certs\") pod \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " Nov 02 14:22:45.193671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.192204 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-podinfo\") pod \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " Nov 02 14:22:45.193671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.192270 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-kube-api-access-2s29r\" (UniqueName: \"kubernetes.io/projected/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-kube-api-access-2s29r\") pod \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " Nov 02 14:22:45.193671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.192305 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-data" pod "8ac034d4-7e4f-468d-8a4d-b712f4d81bae" (UID: "8ac034d4-7e4f-468d-8a4d-b712f4d81bae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:45.193671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.192326 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-envoy\") pod \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " Nov 02 14:22:45.193671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.192364 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-credential-socket\") pod \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " Nov 02 14:22:45.193671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.192447 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-workload-socket\") pod \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " Nov 02 14:22:45.193671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.192498 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istiod-ca-cert\") pod \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " Nov 02 14:22:45.193671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.192565 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-token\" (UniqueName: \"kubernetes.io/projected/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-token\") pod \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\" (UID: \"8ac034d4-7e4f-468d-8a4d-b712f4d81bae\") " Nov 02 14:22:45.193671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.192861 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-credential-socket" pod "8ac034d4-7e4f-468d-8a4d-b712f4d81bae" (UID: "8ac034d4-7e4f-468d-8a4d-b712f4d81bae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:45.193671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.193364 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-workload-certs" pod "8ac034d4-7e4f-468d-8a4d-b712f4d81bae" (UID: "8ac034d4-7e4f-468d-8a4d-b712f4d81bae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:45.193671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.193572 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:45.193671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.193594 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:45.193671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.193608 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:45.194691 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.193732 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-workload-socket" pod "8ac034d4-7e4f-468d-8a4d-b712f4d81bae" (UID: "8ac034d4-7e4f-468d-8a4d-b712f4d81bae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:45.195147 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.195106 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istiod-ca-cert" pod "8ac034d4-7e4f-468d-8a4d-b712f4d81bae" (UID: "8ac034d4-7e4f-468d-8a4d-b712f4d81bae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:45.197402 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.197372 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-envoy" pod "8ac034d4-7e4f-468d-8a4d-b712f4d81bae" (UID: "8ac034d4-7e4f-468d-8a4d-b712f4d81bae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:45.199369 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.199343 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-podinfo" pod "8ac034d4-7e4f-468d-8a4d-b712f4d81bae" (UID: "8ac034d4-7e4f-468d-8a4d-b712f4d81bae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:22:45.199472 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.199347 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-token" pod "8ac034d4-7e4f-468d-8a4d-b712f4d81bae" (UID: "8ac034d4-7e4f-468d-8a4d-b712f4d81bae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:45.201436 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.201411 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-kube-api-access-2s29r" pod "8ac034d4-7e4f-468d-8a4d-b712f4d81bae" (UID: "8ac034d4-7e4f-468d-8a4d-b712f4d81bae"). InnerVolumeSpecName "kube-api-access-2s29r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:45.230821 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.230787 4866 generic.go:356] "Generic (PLEG): container finished" podID="8ac034d4-7e4f-468d-8a4d-b712f4d81bae" containerID="fe245455cbf9c36166526f4b712cb01d1d7bc73d40e473db06c2f9322aae81fa" exitCode=0 Nov 02 14:22:45.231025 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.230839 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" Nov 02 14:22:45.231025 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.230879 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" event={"ID":"8ac034d4-7e4f-468d-8a4d-b712f4d81bae","Type":"ContainerDied","Data":"fe245455cbf9c36166526f4b712cb01d1d7bc73d40e473db06c2f9322aae81fa"} Nov 02 14:22:45.231025 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.230989 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg" event={"ID":"8ac034d4-7e4f-468d-8a4d-b712f4d81bae","Type":"ContainerDied","Data":"15869d1c629c106ee6468e842badb7178121cba4c315967855a655c8494494f4"} Nov 02 14:22:45.231205 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.231026 4866 scope.go:122] "RemoveContainer" containerID="fe245455cbf9c36166526f4b712cb01d1d7bc73d40e473db06c2f9322aae81fa" Nov 02 14:22:45.234909 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.234882 4866 generic.go:356] "Generic (PLEG): container finished" podID="65cbb15a-d104-4b88-9147-290d212c6842" containerID="d533a9420cde800e9a72a49b5c82a9a2b1b9f41195bba232db215b97b2055188" exitCode=0 Nov 02 14:22:45.235151 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.235128 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" event={"ID":"65cbb15a-d104-4b88-9147-290d212c6842","Type":"ContainerDied","Data":"d533a9420cde800e9a72a49b5c82a9a2b1b9f41195bba232db215b97b2055188"} Nov 02 14:22:45.237910 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.237889 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:22:45.246327 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.246305 4866 scope.go:122] "RemoveContainer" containerID="fe245455cbf9c36166526f4b712cb01d1d7bc73d40e473db06c2f9322aae81fa" Nov 02 14:22:45.246678 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:22:45.246653 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe245455cbf9c36166526f4b712cb01d1d7bc73d40e473db06c2f9322aae81fa\": container with ID starting with fe245455cbf9c36166526f4b712cb01d1d7bc73d40e473db06c2f9322aae81fa not found: ID does not exist" containerID="fe245455cbf9c36166526f4b712cb01d1d7bc73d40e473db06c2f9322aae81fa" Nov 02 14:22:45.246734 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.246694 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe245455cbf9c36166526f4b712cb01d1d7bc73d40e473db06c2f9322aae81fa"} err="failed to get container status \"fe245455cbf9c36166526f4b712cb01d1d7bc73d40e473db06c2f9322aae81fa\": rpc error: code = NotFound desc = could not find container \"fe245455cbf9c36166526f4b712cb01d1d7bc73d40e473db06c2f9322aae81fa\": container with ID starting with fe245455cbf9c36166526f4b712cb01d1d7bc73d40e473db06c2f9322aae81fa not found: ID does not exist" Nov 02 14:22:45.295045 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.295009 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-workload-certs\") pod \"65cbb15a-d104-4b88-9147-290d212c6842\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " Nov 02 14:22:45.295243 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.295075 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-istio-envoy\") pod \"65cbb15a-d104-4b88-9147-290d212c6842\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " Nov 02 14:22:45.295243 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.295166 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-workload-socket\") pod \"65cbb15a-d104-4b88-9147-290d212c6842\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " Nov 02 14:22:45.295243 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.295233 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65cbb15a-d104-4b88-9147-290d212c6842-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65cbb15a-d104-4b88-9147-290d212c6842-istio-podinfo\") pod \"65cbb15a-d104-4b88-9147-290d212c6842\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " Nov 02 14:22:45.295409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.295261 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65cbb15a-d104-4b88-9147-290d212c6842-kube-api-access-j94sq\" (UniqueName: \"kubernetes.io/projected/65cbb15a-d104-4b88-9147-290d212c6842-kube-api-access-j94sq\") pod \"65cbb15a-d104-4b88-9147-290d212c6842\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " Nov 02 14:22:45.295409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.295325 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-istio-data\") pod \"65cbb15a-d104-4b88-9147-290d212c6842\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " Nov 02 14:22:45.295409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.295360 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-workload-certs" pod "65cbb15a-d104-4b88-9147-290d212c6842" (UID: "65cbb15a-d104-4b88-9147-290d212c6842"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:45.295409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.295405 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65cbb15a-d104-4b88-9147-290d212c6842-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65cbb15a-d104-4b88-9147-290d212c6842-istiod-ca-cert\") pod \"65cbb15a-d104-4b88-9147-290d212c6842\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " Nov 02 14:22:45.295616 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.295432 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65cbb15a-d104-4b88-9147-290d212c6842-istio-token\" (UniqueName: \"kubernetes.io/projected/65cbb15a-d104-4b88-9147-290d212c6842-istio-token\") pod \"65cbb15a-d104-4b88-9147-290d212c6842\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " Nov 02 14:22:45.295616 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.295571 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-credential-socket\") pod \"65cbb15a-d104-4b88-9147-290d212c6842\" (UID: \"65cbb15a-d104-4b88-9147-290d212c6842\") " Nov 02 14:22:45.295712 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.295656 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-istio-data" pod "65cbb15a-d104-4b88-9147-290d212c6842" (UID: "65cbb15a-d104-4b88-9147-290d212c6842"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:45.296718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.296089 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-workload-socket" pod "65cbb15a-d104-4b88-9147-290d212c6842" (UID: "65cbb15a-d104-4b88-9147-290d212c6842"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:45.296718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.296136 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:45.296718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.296157 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:45.296718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.296172 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s29r\" (UniqueName: \"kubernetes.io/projected/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-kube-api-access-2s29r\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:45.296718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.296189 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:45.296718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.296204 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:45.296718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.296218 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:45.296718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.296234 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:45.296718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.296247 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac034d4-7e4f-468d-8a4d-b712f4d81bae-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:45.296718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.296627 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65cbb15a-d104-4b88-9147-290d212c6842-istiod-ca-cert" pod "65cbb15a-d104-4b88-9147-290d212c6842" (UID: "65cbb15a-d104-4b88-9147-290d212c6842"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:45.296718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.296692 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-credential-socket" pod "65cbb15a-d104-4b88-9147-290d212c6842" (UID: "65cbb15a-d104-4b88-9147-290d212c6842"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:45.297912 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.297879 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-istio-envoy" pod "65cbb15a-d104-4b88-9147-290d212c6842" (UID: "65cbb15a-d104-4b88-9147-290d212c6842"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:45.298037 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.298014 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65cbb15a-d104-4b88-9147-290d212c6842-istio-podinfo" pod "65cbb15a-d104-4b88-9147-290d212c6842" (UID: "65cbb15a-d104-4b88-9147-290d212c6842"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:22:45.298369 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.298346 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65cbb15a-d104-4b88-9147-290d212c6842-istio-token" pod "65cbb15a-d104-4b88-9147-290d212c6842" (UID: "65cbb15a-d104-4b88-9147-290d212c6842"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:45.298565 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.298522 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65cbb15a-d104-4b88-9147-290d212c6842-kube-api-access-j94sq" pod "65cbb15a-d104-4b88-9147-290d212c6842" (UID: "65cbb15a-d104-4b88-9147-290d212c6842"). InnerVolumeSpecName "kube-api-access-j94sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:45.331050 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.331012 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg"] Nov 02 14:22:45.337237 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.337183 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4fmpg"] Nov 02 14:22:45.398001 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.397890 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65cbb15a-d104-4b88-9147-290d212c6842-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:45.398001 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.397921 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j94sq\" (UniqueName: \"kubernetes.io/projected/65cbb15a-d104-4b88-9147-290d212c6842-kube-api-access-j94sq\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:45.398001 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.397933 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65cbb15a-d104-4b88-9147-290d212c6842-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:45.398001 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.397975 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65cbb15a-d104-4b88-9147-290d212c6842-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:45.398001 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.397983 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:45.398001 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.397991 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:45.398001 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:45.398002 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cbb15a-d104-4b88-9147-290d212c6842-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:22:46.244178 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:46.244144 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" Nov 02 14:22:46.244634 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:46.244138 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8" event={"ID":"65cbb15a-d104-4b88-9147-290d212c6842","Type":"ContainerDied","Data":"5c268f2e0cf5e6ea1b561ec8025e5e5817e74c0105d439fa7f02bca87740a653"} Nov 02 14:22:46.244634 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:46.244315 4866 scope.go:122] "RemoveContainer" containerID="d533a9420cde800e9a72a49b5c82a9a2b1b9f41195bba232db215b97b2055188" Nov 02 14:22:46.309076 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:46.309045 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8"] Nov 02 14:22:46.320585 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:46.320552 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-z4nx8"] Nov 02 14:22:46.947316 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:46.947282 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65cbb15a-d104-4b88-9147-290d212c6842" path="/var/lib/kubelet/pods/65cbb15a-d104-4b88-9147-290d212c6842/volumes" Nov 02 14:22:46.947837 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:46.947821 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac034d4-7e4f-468d-8a4d-b712f4d81bae" path="/var/lib/kubelet/pods/8ac034d4-7e4f-468d-8a4d-b712f4d81bae/volumes" Nov 02 14:22:54.036677 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.036638 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j"] Nov 02 14:22:54.037271 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.037255 4866 cpu_manager.go:414] "containerMap: removing container" podUID="747be1e1-d76f-4d33-bb59-1e8eb9af3c98" containerName="payload-processing" Nov 02 14:22:54.037315 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.037272 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747be1e1-d76f-4d33-bb59-1e8eb9af3c98" containerName="payload-processing" Nov 02 14:22:54.037315 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.037286 4866 cpu_manager.go:414] "containerMap: removing container" podUID="65cbb15a-d104-4b88-9147-290d212c6842" containerName="istio-proxy" Nov 02 14:22:54.037315 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.037291 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65cbb15a-d104-4b88-9147-290d212c6842" containerName="istio-proxy" Nov 02 14:22:54.037315 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.037302 4866 cpu_manager.go:414] "containerMap: removing container" podUID="8ac034d4-7e4f-468d-8a4d-b712f4d81bae" containerName="istio-proxy" Nov 02 14:22:54.037315 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.037309 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac034d4-7e4f-468d-8a4d-b712f4d81bae" containerName="istio-proxy" Nov 02 14:22:54.037466 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.037319 4866 cpu_manager.go:414] "containerMap: removing container" podUID="cf657ad7-55b6-4568-8811-179057527b28" containerName="payload-processing" Nov 02 14:22:54.037466 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.037324 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf657ad7-55b6-4568-8811-179057527b28" containerName="payload-processing" Nov 02 14:22:54.037466 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.037409 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac034d4-7e4f-468d-8a4d-b712f4d81bae" containerName="istio-proxy" Nov 02 14:22:54.037466 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.037417 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf657ad7-55b6-4568-8811-179057527b28" containerName="payload-processing" Nov 02 14:22:54.037466 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.037428 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="65cbb15a-d104-4b88-9147-290d212c6842" containerName="istio-proxy" Nov 02 14:22:54.037466 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.037434 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="747be1e1-d76f-4d33-bb59-1e8eb9af3c98" containerName="payload-processing" Nov 02 14:22:54.042740 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.042719 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" Nov 02 14:22:54.062012 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.061985 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-944506\"" Nov 02 14:22:54.065382 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.065351 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j"] Nov 02 14:22:54.073940 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.073914 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-944506-dockercfg-8xzbl\"" Nov 02 14:22:54.134891 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.134685 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh7cl\" (UniqueName: \"kubernetes.io/projected/7ff352ba-6b9c-4cb7-94d6-efc1de88be37-kube-api-access-lh7cl\") pod \"payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j\" (UID: \"7ff352ba-6b9c-4cb7-94d6-efc1de88be37\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" Nov 02 14:22:54.136164 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.135490 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff352ba-6b9c-4cb7-94d6-efc1de88be37-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j\" (UID: \"7ff352ba-6b9c-4cb7-94d6-efc1de88be37\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" Nov 02 14:22:54.139075 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.138928 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w"] Nov 02 14:22:54.148668 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.148638 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" Nov 02 14:22:54.184384 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.181442 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w"] Nov 02 14:22:54.255577 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.255535 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e67ad2-0fb8-4025-ac30-8ea39ef94f51-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w\" (UID: \"b6e67ad2-0fb8-4025-ac30-8ea39ef94f51\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" Nov 02 14:22:54.257746 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.257705 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh7cl\" (UniqueName: \"kubernetes.io/projected/7ff352ba-6b9c-4cb7-94d6-efc1de88be37-kube-api-access-lh7cl\") pod \"payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j\" (UID: \"7ff352ba-6b9c-4cb7-94d6-efc1de88be37\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" Nov 02 14:22:54.257911 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.257817 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff352ba-6b9c-4cb7-94d6-efc1de88be37-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j\" (UID: \"7ff352ba-6b9c-4cb7-94d6-efc1de88be37\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" Nov 02 14:22:54.257911 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.257847 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxfzv\" (UniqueName: \"kubernetes.io/projected/b6e67ad2-0fb8-4025-ac30-8ea39ef94f51-kube-api-access-dxfzv\") pod \"payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w\" (UID: \"b6e67ad2-0fb8-4025-ac30-8ea39ef94f51\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" Nov 02 14:22:54.258910 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.258884 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff352ba-6b9c-4cb7-94d6-efc1de88be37-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j\" (UID: \"7ff352ba-6b9c-4cb7-94d6-efc1de88be37\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" Nov 02 14:22:54.291059 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.290283 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh7cl\" (UniqueName: \"kubernetes.io/projected/7ff352ba-6b9c-4cb7-94d6-efc1de88be37-kube-api-access-lh7cl\") pod \"payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j\" (UID: \"7ff352ba-6b9c-4cb7-94d6-efc1de88be37\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" Nov 02 14:22:54.356423 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.356385 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" Nov 02 14:22:54.359059 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.359026 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxfzv\" (UniqueName: \"kubernetes.io/projected/b6e67ad2-0fb8-4025-ac30-8ea39ef94f51-kube-api-access-dxfzv\") pod \"payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w\" (UID: \"b6e67ad2-0fb8-4025-ac30-8ea39ef94f51\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" Nov 02 14:22:54.359198 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.359167 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e67ad2-0fb8-4025-ac30-8ea39ef94f51-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w\" (UID: \"b6e67ad2-0fb8-4025-ac30-8ea39ef94f51\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" Nov 02 14:22:54.360055 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.359837 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e67ad2-0fb8-4025-ac30-8ea39ef94f51-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w\" (UID: \"b6e67ad2-0fb8-4025-ac30-8ea39ef94f51\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" Nov 02 14:22:54.385405 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.385370 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxfzv\" (UniqueName: \"kubernetes.io/projected/b6e67ad2-0fb8-4025-ac30-8ea39ef94f51-kube-api-access-dxfzv\") pod \"payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w\" (UID: \"b6e67ad2-0fb8-4025-ac30-8ea39ef94f51\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" Nov 02 14:22:54.489331 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.489286 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" Nov 02 14:22:54.663649 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.663621 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j"] Nov 02 14:22:54.664958 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:22:54.664920 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff352ba_6b9c_4cb7_94d6_efc1de88be37.slice/crio-df04d372aa99d348552eced2cc31804908f7c05c7eca61a0c6bf119248825fed WatchSource:0}: Error finding container df04d372aa99d348552eced2cc31804908f7c05c7eca61a0c6bf119248825fed: Status 404 returned error can't find the container with id df04d372aa99d348552eced2cc31804908f7c05c7eca61a0c6bf119248825fed Nov 02 14:22:54.768432 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.768400 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk"] Nov 02 14:22:54.768620 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.768600 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" podUID="4e4659ae-3de7-4902-9fdf-98373b40172c" containerName="istio-proxy" containerID="cri-o://63d0196c9c9e8594c6a3e4c934fc9da87fab1f46286cb7ce053de3c4e74cdd87" gracePeriod=30 Nov 02 14:22:54.799431 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.799350 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w"] Nov 02 14:22:54.803990 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:22:54.803919 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e67ad2_0fb8_4025_ac30_8ea39ef94f51.slice/crio-6b86d00ee0b5a7a55fdbb67380bbad454da4a4104c8a98e0716d856a2b5245a0 WatchSource:0}: Error finding container 6b86d00ee0b5a7a55fdbb67380bbad454da4a4104c8a98e0716d856a2b5245a0: Status 404 returned error can't find the container with id 6b86d00ee0b5a7a55fdbb67380bbad454da4a4104c8a98e0716d856a2b5245a0 Nov 02 14:22:54.908344 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.908308 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4"] Nov 02 14:22:54.908602 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:54.908577 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" podUID="b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" containerName="istio-proxy" containerID="cri-o://3497d0eab8d1b5a104e6e248bf1e6118ed156cc7fd7f3ed8122f75557ef6ae1c" gracePeriod=30 Nov 02 14:22:55.308549 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:55.308513 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" event={"ID":"7ff352ba-6b9c-4cb7-94d6-efc1de88be37","Type":"ContainerStarted","Data":"9d8da50c6d06f351ab137e68728c9ccba8ae4d5bf8f44fbbb1f7324956a8a698"} Nov 02 14:22:55.309012 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:55.308571 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" event={"ID":"7ff352ba-6b9c-4cb7-94d6-efc1de88be37","Type":"ContainerStarted","Data":"df04d372aa99d348552eced2cc31804908f7c05c7eca61a0c6bf119248825fed"} Nov 02 14:22:55.309012 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:55.308619 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" Nov 02 14:22:55.310646 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:55.310615 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" event={"ID":"b6e67ad2-0fb8-4025-ac30-8ea39ef94f51","Type":"ContainerStarted","Data":"51c129b40af48f58584ab3674fd94ac0a5ba8659f1a56ab875c894594f7baa67"} Nov 02 14:22:55.310783 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:55.310670 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" event={"ID":"b6e67ad2-0fb8-4025-ac30-8ea39ef94f51","Type":"ContainerStarted","Data":"6b86d00ee0b5a7a55fdbb67380bbad454da4a4104c8a98e0716d856a2b5245a0"} Nov 02 14:22:55.310783 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:55.310698 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" Nov 02 14:22:55.352727 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:55.352665 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" podStartSLOduration=1.352651144 podStartE2EDuration="1.352651144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:55.349075196 +0000 UTC m=+1540.532922534" watchObservedRunningTime="2026-11-02 14:22:55.352651144 +0000 UTC m=+1540.536498482" Nov 02 14:22:55.390359 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:22:55.390307 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" podStartSLOduration=1.390292746 podStartE2EDuration="1.390292746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:55.389172676 +0000 UTC m=+1540.573020013" watchObservedRunningTime="2026-11-02 14:22:55.390292746 +0000 UTC m=+1540.574140095" Nov 02 14:23:00.045359 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.045279 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:23:00.151327 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.151288 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-credential-socket\") pod \"4e4659ae-3de7-4902-9fdf-98373b40172c\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " Nov 02 14:23:00.151522 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.151369 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-envoy\") pod \"4e4659ae-3de7-4902-9fdf-98373b40172c\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " Nov 02 14:23:00.151522 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.151414 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-token\" (UniqueName: \"kubernetes.io/projected/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-token\") pod \"4e4659ae-3de7-4902-9fdf-98373b40172c\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " Nov 02 14:23:00.151522 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.151461 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-podinfo\") pod \"4e4659ae-3de7-4902-9fdf-98373b40172c\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " Nov 02 14:23:00.151682 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.151548 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-workload-socket\") pod \"4e4659ae-3de7-4902-9fdf-98373b40172c\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " Nov 02 14:23:00.151682 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.151630 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-credential-socket" pod "4e4659ae-3de7-4902-9fdf-98373b40172c" (UID: "4e4659ae-3de7-4902-9fdf-98373b40172c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:00.151682 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.151649 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-workload-certs\") pod \"4e4659ae-3de7-4902-9fdf-98373b40172c\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " Nov 02 14:23:00.151682 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.151677 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4659ae-3de7-4902-9fdf-98373b40172c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4659ae-3de7-4902-9fdf-98373b40172c-istiod-ca-cert\") pod \"4e4659ae-3de7-4902-9fdf-98373b40172c\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " Nov 02 14:23:00.151883 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.151724 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4659ae-3de7-4902-9fdf-98373b40172c-kube-api-access-276gh\" (UniqueName: \"kubernetes.io/projected/4e4659ae-3de7-4902-9fdf-98373b40172c-kube-api-access-276gh\") pod \"4e4659ae-3de7-4902-9fdf-98373b40172c\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " Nov 02 14:23:00.151883 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.151759 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-data\") pod \"4e4659ae-3de7-4902-9fdf-98373b40172c\" (UID: \"4e4659ae-3de7-4902-9fdf-98373b40172c\") " Nov 02 14:23:00.151883 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.151872 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-workload-socket" pod "4e4659ae-3de7-4902-9fdf-98373b40172c" (UID: "4e4659ae-3de7-4902-9fdf-98373b40172c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:00.153011 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.152118 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-workload-certs" pod "4e4659ae-3de7-4902-9fdf-98373b40172c" (UID: "4e4659ae-3de7-4902-9fdf-98373b40172c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:00.153011 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.152373 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:00.153011 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.152380 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-data" pod "4e4659ae-3de7-4902-9fdf-98373b40172c" (UID: "4e4659ae-3de7-4902-9fdf-98373b40172c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:00.153011 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.152394 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:00.153011 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.152409 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:00.153011 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.152729 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4659ae-3de7-4902-9fdf-98373b40172c-istiod-ca-cert" pod "4e4659ae-3de7-4902-9fdf-98373b40172c" (UID: "4e4659ae-3de7-4902-9fdf-98373b40172c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:23:00.155409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.155325 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-token" pod "4e4659ae-3de7-4902-9fdf-98373b40172c" (UID: "4e4659ae-3de7-4902-9fdf-98373b40172c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:23:00.155471 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.155413 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4659ae-3de7-4902-9fdf-98373b40172c-kube-api-access-276gh" pod "4e4659ae-3de7-4902-9fdf-98373b40172c" (UID: "4e4659ae-3de7-4902-9fdf-98373b40172c"). InnerVolumeSpecName "kube-api-access-276gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:23:00.155555 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.155524 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-podinfo" pod "4e4659ae-3de7-4902-9fdf-98373b40172c" (UID: "4e4659ae-3de7-4902-9fdf-98373b40172c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:23:00.156270 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.156248 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-envoy" pod "4e4659ae-3de7-4902-9fdf-98373b40172c" (UID: "4e4659ae-3de7-4902-9fdf-98373b40172c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:00.179031 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.179004 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:23:00.253339 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.253300 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-token\" (UniqueName: \"kubernetes.io/projected/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-token\") pod \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " Nov 02 14:23:00.253540 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.253380 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istiod-ca-cert\") pod \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " Nov 02 14:23:00.253540 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.253491 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-kube-api-access-n29mg\" (UniqueName: \"kubernetes.io/projected/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-kube-api-access-n29mg\") pod \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " Nov 02 14:23:00.253540 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.253526 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-envoy\") pod \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " Nov 02 14:23:00.253723 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.253559 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-podinfo\") pod \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " Nov 02 14:23:00.253723 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.253596 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-workload-certs\") pod \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " Nov 02 14:23:00.253723 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.253703 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-credential-socket\") pod \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " Nov 02 14:23:00.253873 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.253753 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-data\") pod \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " Nov 02 14:23:00.253925 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.253877 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-workload-socket\") pod \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\" (UID: \"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910\") " Nov 02 14:23:00.254189 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.254160 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istiod-ca-cert" pod "b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" (UID: "b7ebf7e9-7f0f-4489-893d-d2ad6efc1910"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:23:00.254413 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.254383 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-workload-certs" pod "b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" (UID: "b7ebf7e9-7f0f-4489-893d-d2ad6efc1910"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:00.254557 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.254520 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:00.254557 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.254538 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:00.254718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.254552 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:00.254718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.254599 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:00.254718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.254613 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:00.254718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.254627 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4659ae-3de7-4902-9fdf-98373b40172c-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:00.254718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.254642 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-276gh\" (UniqueName: \"kubernetes.io/projected/4e4659ae-3de7-4902-9fdf-98373b40172c-kube-api-access-276gh\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:00.254718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.254655 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4659ae-3de7-4902-9fdf-98373b40172c-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:00.254718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.254619 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-workload-socket" pod "b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" (UID: "b7ebf7e9-7f0f-4489-893d-d2ad6efc1910"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:00.254718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.254632 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-credential-socket" pod "b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" (UID: "b7ebf7e9-7f0f-4489-893d-d2ad6efc1910"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:00.254718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.254688 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-data" pod "b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" (UID: "b7ebf7e9-7f0f-4489-893d-d2ad6efc1910"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:00.256269 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.256241 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-token" pod "b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" (UID: "b7ebf7e9-7f0f-4489-893d-d2ad6efc1910"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:23:00.256807 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.256775 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-podinfo" pod "b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" (UID: "b7ebf7e9-7f0f-4489-893d-d2ad6efc1910"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:23:00.256914 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.256877 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-envoy" pod "b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" (UID: "b7ebf7e9-7f0f-4489-893d-d2ad6efc1910"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:00.257498 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.257473 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-kube-api-access-n29mg" pod "b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" (UID: "b7ebf7e9-7f0f-4489-893d-d2ad6efc1910"). InnerVolumeSpecName "kube-api-access-n29mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:23:00.343055 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.342965 4866 generic.go:356] "Generic (PLEG): container finished" podID="4e4659ae-3de7-4902-9fdf-98373b40172c" containerID="63d0196c9c9e8594c6a3e4c934fc9da87fab1f46286cb7ce053de3c4e74cdd87" exitCode=0 Nov 02 14:23:00.343055 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.343020 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" Nov 02 14:23:00.343055 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.343036 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" event={"ID":"4e4659ae-3de7-4902-9fdf-98373b40172c","Type":"ContainerDied","Data":"63d0196c9c9e8594c6a3e4c934fc9da87fab1f46286cb7ce053de3c4e74cdd87"} Nov 02 14:23:00.343303 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.343101 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk" event={"ID":"4e4659ae-3de7-4902-9fdf-98373b40172c","Type":"ContainerDied","Data":"8751718059de7449307ce25d44a289eb72115852a6130eb276708d9861ca349e"} Nov 02 14:23:00.343303 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.343129 4866 scope.go:122] "RemoveContainer" containerID="63d0196c9c9e8594c6a3e4c934fc9da87fab1f46286cb7ce053de3c4e74cdd87" Nov 02 14:23:00.344832 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.344804 4866 generic.go:356] "Generic (PLEG): container finished" podID="b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" containerID="3497d0eab8d1b5a104e6e248bf1e6118ed156cc7fd7f3ed8122f75557ef6ae1c" exitCode=0 Nov 02 14:23:00.344931 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.344867 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" Nov 02 14:23:00.344931 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.344893 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" event={"ID":"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910","Type":"ContainerDied","Data":"3497d0eab8d1b5a104e6e248bf1e6118ed156cc7fd7f3ed8122f75557ef6ae1c"} Nov 02 14:23:00.345071 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.344982 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4" event={"ID":"b7ebf7e9-7f0f-4489-893d-d2ad6efc1910","Type":"ContainerDied","Data":"d9ceb0544315a10f31cca5bde5f9db7fd738761d1202f65f79561e036acfc146"} Nov 02 14:23:00.353939 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.353917 4866 scope.go:122] "RemoveContainer" containerID="63d0196c9c9e8594c6a3e4c934fc9da87fab1f46286cb7ce053de3c4e74cdd87" Nov 02 14:23:00.354421 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:23:00.354344 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d0196c9c9e8594c6a3e4c934fc9da87fab1f46286cb7ce053de3c4e74cdd87\": container with ID starting with 63d0196c9c9e8594c6a3e4c934fc9da87fab1f46286cb7ce053de3c4e74cdd87 not found: ID does not exist" containerID="63d0196c9c9e8594c6a3e4c934fc9da87fab1f46286cb7ce053de3c4e74cdd87" Nov 02 14:23:00.354421 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.354384 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d0196c9c9e8594c6a3e4c934fc9da87fab1f46286cb7ce053de3c4e74cdd87"} err="failed to get container status \"63d0196c9c9e8594c6a3e4c934fc9da87fab1f46286cb7ce053de3c4e74cdd87\": rpc error: code = NotFound desc = could not find container \"63d0196c9c9e8594c6a3e4c934fc9da87fab1f46286cb7ce053de3c4e74cdd87\": container with ID starting with 63d0196c9c9e8594c6a3e4c934fc9da87fab1f46286cb7ce053de3c4e74cdd87 not found: ID does not exist" Nov 02 14:23:00.354421 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.354411 4866 scope.go:122] "RemoveContainer" containerID="3497d0eab8d1b5a104e6e248bf1e6118ed156cc7fd7f3ed8122f75557ef6ae1c" Nov 02 14:23:00.355560 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.355540 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:00.355640 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.355566 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:00.355640 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.355582 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n29mg\" (UniqueName: \"kubernetes.io/projected/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-kube-api-access-n29mg\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:00.355640 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.355597 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:00.355640 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.355610 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:00.355640 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.355623 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:00.355797 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.355634 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:00.364045 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.364028 4866 scope.go:122] "RemoveContainer" containerID="f57d6ee3c1381a21cf89a46a89b45153435a3f9c8285aeeaf8dfc37387b55e8b" Nov 02 14:23:00.373049 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.373022 4866 scope.go:122] "RemoveContainer" containerID="3497d0eab8d1b5a104e6e248bf1e6118ed156cc7fd7f3ed8122f75557ef6ae1c" Nov 02 14:23:00.373368 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:23:00.373348 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3497d0eab8d1b5a104e6e248bf1e6118ed156cc7fd7f3ed8122f75557ef6ae1c\": container with ID starting with 3497d0eab8d1b5a104e6e248bf1e6118ed156cc7fd7f3ed8122f75557ef6ae1c not found: ID does not exist" containerID="3497d0eab8d1b5a104e6e248bf1e6118ed156cc7fd7f3ed8122f75557ef6ae1c" Nov 02 14:23:00.373437 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.373379 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3497d0eab8d1b5a104e6e248bf1e6118ed156cc7fd7f3ed8122f75557ef6ae1c"} err="failed to get container status \"3497d0eab8d1b5a104e6e248bf1e6118ed156cc7fd7f3ed8122f75557ef6ae1c\": rpc error: code = NotFound desc = could not find container \"3497d0eab8d1b5a104e6e248bf1e6118ed156cc7fd7f3ed8122f75557ef6ae1c\": container with ID starting with 3497d0eab8d1b5a104e6e248bf1e6118ed156cc7fd7f3ed8122f75557ef6ae1c not found: ID does not exist" Nov 02 14:23:00.373437 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.373399 4866 scope.go:122] "RemoveContainer" containerID="f57d6ee3c1381a21cf89a46a89b45153435a3f9c8285aeeaf8dfc37387b55e8b" Nov 02 14:23:00.373647 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:23:00.373629 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57d6ee3c1381a21cf89a46a89b45153435a3f9c8285aeeaf8dfc37387b55e8b\": container with ID starting with f57d6ee3c1381a21cf89a46a89b45153435a3f9c8285aeeaf8dfc37387b55e8b not found: ID does not exist" containerID="f57d6ee3c1381a21cf89a46a89b45153435a3f9c8285aeeaf8dfc37387b55e8b" Nov 02 14:23:00.373700 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.373651 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57d6ee3c1381a21cf89a46a89b45153435a3f9c8285aeeaf8dfc37387b55e8b"} err="failed to get container status \"f57d6ee3c1381a21cf89a46a89b45153435a3f9c8285aeeaf8dfc37387b55e8b\": rpc error: code = NotFound desc = could not find container \"f57d6ee3c1381a21cf89a46a89b45153435a3f9c8285aeeaf8dfc37387b55e8b\": container with ID starting with f57d6ee3c1381a21cf89a46a89b45153435a3f9c8285aeeaf8dfc37387b55e8b not found: ID does not exist" Nov 02 14:23:00.400440 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.400400 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk"] Nov 02 14:23:00.403872 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.403842 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-qtlwk"] Nov 02 14:23:00.436925 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.436890 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4"] Nov 02 14:23:00.442602 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.442573 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-4jgf4"] Nov 02 14:23:00.946705 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.946667 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4659ae-3de7-4902-9fdf-98373b40172c" path="/var/lib/kubelet/pods/4e4659ae-3de7-4902-9fdf-98373b40172c/volumes" Nov 02 14:23:00.947505 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:00.947474 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" path="/var/lib/kubelet/pods/b7ebf7e9-7f0f-4489-893d-d2ad6efc1910/volumes" Nov 02 14:23:05.232795 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.232765 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-747f596d58-lncqr"] Nov 02 14:23:05.233381 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.233365 4866 cpu_manager.go:414] "containerMap: removing container" podUID="b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" containerName="istio-proxy" Nov 02 14:23:05.233427 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.233382 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" containerName="istio-proxy" Nov 02 14:23:05.233427 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.233403 4866 cpu_manager.go:414] "containerMap: removing container" podUID="4e4659ae-3de7-4902-9fdf-98373b40172c" containerName="istio-proxy" Nov 02 14:23:05.233427 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.233407 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4659ae-3de7-4902-9fdf-98373b40172c" containerName="istio-proxy" Nov 02 14:23:05.233427 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.233419 4866 cpu_manager.go:414] "containerMap: removing container" podUID="b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" containerName="istio-proxy" Nov 02 14:23:05.233427 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.233424 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" containerName="istio-proxy" Nov 02 14:23:05.233633 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.233499 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" containerName="istio-proxy" Nov 02 14:23:05.233633 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.233513 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4659ae-3de7-4902-9fdf-98373b40172c" containerName="istio-proxy" Nov 02 14:23:05.238901 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.238883 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" Nov 02 14:23:05.257640 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.257611 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-747f596d58-lncqr"] Nov 02 14:23:05.265879 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.265834 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 14:23:05.266036 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.265895 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 14:23:05.266112 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.266056 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 14:23:05.266170 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.266124 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1b98b233190d\"" Nov 02 14:23:05.266223 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.266212 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 14:23:05.268166 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.268150 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fa41858d6534\"" Nov 02 14:23:05.268270 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.268209 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-56zdx\"" Nov 02 14:23:05.319941 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.319877 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn4fd\" (UniqueName: \"kubernetes.io/projected/3126935b-ac99-4227-b77d-b54707cd3dba-kube-api-access-jn4fd\") pod \"payload-processing-external-model-747f596d58-lncqr\" (UID: \"3126935b-ac99-4227-b77d-b54707cd3dba\") " pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" Nov 02 14:23:05.320340 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.319931 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3126935b-ac99-4227-b77d-b54707cd3dba-routing-overlay-volume\") pod \"payload-processing-external-model-747f596d58-lncqr\" (UID: \"3126935b-ac99-4227-b77d-b54707cd3dba\") " pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" Nov 02 14:23:05.320340 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.320276 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3126935b-ac99-4227-b77d-b54707cd3dba-provider-credentials\") pod \"payload-processing-external-model-747f596d58-lncqr\" (UID: \"3126935b-ac99-4227-b77d-b54707cd3dba\") " pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" Nov 02 14:23:05.320516 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.320486 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3126935b-ac99-4227-b77d-b54707cd3dba-plugins-config-volume\") pod \"payload-processing-external-model-747f596d58-lncqr\" (UID: \"3126935b-ac99-4227-b77d-b54707cd3dba\") " pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" Nov 02 14:23:05.421937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.421901 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4fd\" (UniqueName: \"kubernetes.io/projected/3126935b-ac99-4227-b77d-b54707cd3dba-kube-api-access-jn4fd\") pod \"payload-processing-external-model-747f596d58-lncqr\" (UID: \"3126935b-ac99-4227-b77d-b54707cd3dba\") " pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" Nov 02 14:23:05.421937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.421964 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3126935b-ac99-4227-b77d-b54707cd3dba-routing-overlay-volume\") pod \"payload-processing-external-model-747f596d58-lncqr\" (UID: \"3126935b-ac99-4227-b77d-b54707cd3dba\") " pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" Nov 02 14:23:05.422433 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.422409 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3126935b-ac99-4227-b77d-b54707cd3dba-provider-credentials\") pod \"payload-processing-external-model-747f596d58-lncqr\" (UID: \"3126935b-ac99-4227-b77d-b54707cd3dba\") " pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" Nov 02 14:23:05.422605 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.422587 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3126935b-ac99-4227-b77d-b54707cd3dba-plugins-config-volume\") pod \"payload-processing-external-model-747f596d58-lncqr\" (UID: \"3126935b-ac99-4227-b77d-b54707cd3dba\") " pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" Nov 02 14:23:05.422679 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.422619 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3126935b-ac99-4227-b77d-b54707cd3dba-routing-overlay-volume\") pod \"payload-processing-external-model-747f596d58-lncqr\" (UID: \"3126935b-ac99-4227-b77d-b54707cd3dba\") " pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" Nov 02 14:23:05.424444 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.424417 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3126935b-ac99-4227-b77d-b54707cd3dba-plugins-config-volume\") pod \"payload-processing-external-model-747f596d58-lncqr\" (UID: \"3126935b-ac99-4227-b77d-b54707cd3dba\") " pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" Nov 02 14:23:05.425773 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.425747 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3126935b-ac99-4227-b77d-b54707cd3dba-provider-credentials\") pod \"payload-processing-external-model-747f596d58-lncqr\" (UID: \"3126935b-ac99-4227-b77d-b54707cd3dba\") " pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" Nov 02 14:23:05.447342 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.447305 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn4fd\" (UniqueName: \"kubernetes.io/projected/3126935b-ac99-4227-b77d-b54707cd3dba-kube-api-access-jn4fd\") pod \"payload-processing-external-model-747f596d58-lncqr\" (UID: \"3126935b-ac99-4227-b77d-b54707cd3dba\") " pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" Nov 02 14:23:05.553313 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.553223 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" Nov 02 14:23:05.866619 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:05.866591 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-747f596d58-lncqr"] Nov 02 14:23:05.868564 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:23:05.868528 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3126935b_ac99_4227_b77d_b54707cd3dba.slice/crio-c1bd48d4bac38b0dd5e6b49b21ca3f162fec7f77c3271a4f432bd100e5e2f551 WatchSource:0}: Error finding container c1bd48d4bac38b0dd5e6b49b21ca3f162fec7f77c3271a4f432bd100e5e2f551: Status 404 returned error can't find the container with id c1bd48d4bac38b0dd5e6b49b21ca3f162fec7f77c3271a4f432bd100e5e2f551 Nov 02 14:23:06.010718 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:23:06.010672 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:23:06.320354 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:06.320322 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" Nov 02 14:23:06.320780 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:06.320385 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" Nov 02 14:23:06.405679 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:06.405640 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" event={"ID":"3126935b-ac99-4227-b77d-b54707cd3dba","Type":"ContainerStarted","Data":"97ec64359093c17c29ac99cecfb03394b53d1291d6ec205169e8e8d7d515294f"} Nov 02 14:23:06.405861 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:06.405696 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" event={"ID":"3126935b-ac99-4227-b77d-b54707cd3dba","Type":"ContainerStarted","Data":"c1bd48d4bac38b0dd5e6b49b21ca3f162fec7f77c3271a4f432bd100e5e2f551"} Nov 02 14:23:06.405861 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:06.405815 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" Nov 02 14:23:06.449606 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:06.449555 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" podStartSLOduration=1.449542822 podStartE2EDuration="1.449542822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:23:06.448513189 +0000 UTC m=+1551.632360545" watchObservedRunningTime="2026-11-02 14:23:06.449542822 +0000 UTC m=+1551.633390159" Nov 02 14:23:08.113172 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:08.113130 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps"] Nov 02 14:23:08.113639 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:08.113362 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" podUID="34a67bee-8e3b-4029-b6e3-6ebae8070ead" containerName="maas-api" containerID="cri-o://23ab9581e067a2de8e8237c9cdf2328aebdf02af6dcca07a45b993b836d98ea3" gracePeriod=30 Nov 02 14:23:08.420257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:08.420224 4866 generic.go:356] "Generic (PLEG): container finished" podID="34a67bee-8e3b-4029-b6e3-6ebae8070ead" containerID="23ab9581e067a2de8e8237c9cdf2328aebdf02af6dcca07a45b993b836d98ea3" exitCode=0 Nov 02 14:23:08.420389 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:08.420296 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" event={"ID":"34a67bee-8e3b-4029-b6e3-6ebae8070ead","Type":"ContainerDied","Data":"23ab9581e067a2de8e8237c9cdf2328aebdf02af6dcca07a45b993b836d98ea3"} Nov 02 14:23:08.420389 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:08.420363 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" event={"ID":"34a67bee-8e3b-4029-b6e3-6ebae8070ead","Type":"ContainerDied","Data":"777243ee7f6717263ab2b891ec8d19b0b19cbf530126b8bd6f1f5bd85c3a13b6"} Nov 02 14:23:08.420389 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:08.420382 4866 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="777243ee7f6717263ab2b891ec8d19b0b19cbf530126b8bd6f1f5bd85c3a13b6" Nov 02 14:23:08.433893 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:08.433870 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" Nov 02 14:23:08.589297 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:08.589269 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a67bee-8e3b-4029-b6e3-6ebae8070ead-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a67bee-8e3b-4029-b6e3-6ebae8070ead-openshift-service-ca\") pod \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\" (UID: \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\") " Nov 02 14:23:08.589476 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:08.589368 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34a67bee-8e3b-4029-b6e3-6ebae8070ead-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34a67bee-8e3b-4029-b6e3-6ebae8070ead-maas-api-tls\") pod \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\" (UID: \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\") " Nov 02 14:23:08.589476 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:08.589412 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a67bee-8e3b-4029-b6e3-6ebae8070ead-kube-api-access-gn2l6\" (UniqueName: \"kubernetes.io/projected/34a67bee-8e3b-4029-b6e3-6ebae8070ead-kube-api-access-gn2l6\") pod \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\" (UID: \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\") " Nov 02 14:23:08.589476 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:08.589439 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34a67bee-8e3b-4029-b6e3-6ebae8070ead-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34a67bee-8e3b-4029-b6e3-6ebae8070ead-metrics-certs\") pod \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\" (UID: \"34a67bee-8e3b-4029-b6e3-6ebae8070ead\") " Nov 02 14:23:08.589993 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:08.589937 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a67bee-8e3b-4029-b6e3-6ebae8070ead-openshift-service-ca" pod "34a67bee-8e3b-4029-b6e3-6ebae8070ead" (UID: "34a67bee-8e3b-4029-b6e3-6ebae8070ead"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:23:08.594696 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:08.594656 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a67bee-8e3b-4029-b6e3-6ebae8070ead-maas-api-tls" pod "34a67bee-8e3b-4029-b6e3-6ebae8070ead" (UID: "34a67bee-8e3b-4029-b6e3-6ebae8070ead"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:23:08.594896 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:08.594869 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a67bee-8e3b-4029-b6e3-6ebae8070ead-metrics-certs" pod "34a67bee-8e3b-4029-b6e3-6ebae8070ead" (UID: "34a67bee-8e3b-4029-b6e3-6ebae8070ead"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:23:08.597202 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:08.597173 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a67bee-8e3b-4029-b6e3-6ebae8070ead-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:08.597324 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:08.597207 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34a67bee-8e3b-4029-b6e3-6ebae8070ead-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:08.597324 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:08.597222 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34a67bee-8e3b-4029-b6e3-6ebae8070ead-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:08.598549 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:08.598523 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a67bee-8e3b-4029-b6e3-6ebae8070ead-kube-api-access-gn2l6" pod "34a67bee-8e3b-4029-b6e3-6ebae8070ead" (UID: "34a67bee-8e3b-4029-b6e3-6ebae8070ead"). InnerVolumeSpecName "kube-api-access-gn2l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:23:08.698329 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:08.698289 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn2l6\" (UniqueName: \"kubernetes.io/projected/34a67bee-8e3b-4029-b6e3-6ebae8070ead-kube-api-access-gn2l6\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:23:09.426302 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:09.426204 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps" Nov 02 14:23:09.481355 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:09.481318 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps"] Nov 02 14:23:09.493098 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:09.493065 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-87a1a06b-849cfdc6cc-cvcps"] Nov 02 14:23:10.522678 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.522645 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj"] Nov 02 14:23:10.523293 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.523276 4866 cpu_manager.go:414] "containerMap: removing container" podUID="34a67bee-8e3b-4029-b6e3-6ebae8070ead" containerName="maas-api" Nov 02 14:23:10.523293 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.523293 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a67bee-8e3b-4029-b6e3-6ebae8070ead" containerName="maas-api" Nov 02 14:23:10.523410 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.523393 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7ebf7e9-7f0f-4489-893d-d2ad6efc1910" containerName="istio-proxy" Nov 02 14:23:10.523410 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.523404 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a67bee-8e3b-4029-b6e3-6ebae8070ead" containerName="maas-api" Nov 02 14:23:10.527133 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.527112 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.546898 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.546872 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8416e2e2-openshift-default-dockercfg-6wz4h\"" Nov 02 14:23:10.562111 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.562081 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj"] Nov 02 14:23:10.626857 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.626820 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-workload-socket\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.627096 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.626886 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-token\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.627096 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.627018 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-data\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.627096 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.627044 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istiod-ca-cert\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.627096 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.627083 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-credential-socket\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.627371 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.627179 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-envoy\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.627371 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.627281 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-workload-certs\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.627477 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.627382 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-podinfo\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.627477 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.627442 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k8tj\" (UniqueName: \"kubernetes.io/projected/950d2085-58ef-47fb-a3f7-ef8cb5430c36-kube-api-access-7k8tj\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.728691 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.728650 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-podinfo\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.728928 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.728740 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k8tj\" (UniqueName: \"kubernetes.io/projected/950d2085-58ef-47fb-a3f7-ef8cb5430c36-kube-api-access-7k8tj\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.728928 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.728838 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-workload-socket\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.728928 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.728889 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-token\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.728928 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.728915 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-data\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.729171 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.728939 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istiod-ca-cert\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.729171 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.729037 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-credential-socket\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.729171 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.729070 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-envoy\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.729171 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.729168 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-workload-certs\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.729759 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.729729 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-workload-certs\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.729882 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.729761 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-workload-socket\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.729882 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.729804 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-credential-socket\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.729882 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.729865 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-data\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.730196 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.730169 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istiod-ca-cert\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.732457 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.732083 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-envoy\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.732457 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.732415 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-podinfo\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.754217 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.754135 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-token\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.754376 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.754249 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k8tj\" (UniqueName: \"kubernetes.io/projected/950d2085-58ef-47fb-a3f7-ef8cb5430c36-kube-api-access-7k8tj\") pod \"e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.842821 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.842738 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:10.958535 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:10.958502 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a67bee-8e3b-4029-b6e3-6ebae8070ead" path="/var/lib/kubelet/pods/34a67bee-8e3b-4029-b6e3-6ebae8070ead/volumes" Nov 02 14:23:11.214513 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:11.214467 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj"] Nov 02 14:23:11.222779 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:23:11.222748 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950d2085_58ef_47fb_a3f7_ef8cb5430c36.slice/crio-2a34ab698800ba2888e5603eb05e4ffa2d35c0717cb6f81aea4e8a2ed47233c6 WatchSource:0}: Error finding container 2a34ab698800ba2888e5603eb05e4ffa2d35c0717cb6f81aea4e8a2ed47233c6: Status 404 returned error can't find the container with id 2a34ab698800ba2888e5603eb05e4ffa2d35c0717cb6f81aea4e8a2ed47233c6 Nov 02 14:23:11.225317 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:11.225287 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:23:11.225393 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:11.225352 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:23:11.225393 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:11.225381 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:23:11.441274 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:11.441232 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" event={"ID":"950d2085-58ef-47fb-a3f7-ef8cb5430c36","Type":"ContainerStarted","Data":"df13c949b4367e951fe1f6fc4d1d553f358e138f3bcb8b5be25fc087398a7d42"} Nov 02 14:23:11.441455 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:11.441283 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" event={"ID":"950d2085-58ef-47fb-a3f7-ef8cb5430c36","Type":"ContainerStarted","Data":"2a34ab698800ba2888e5603eb05e4ffa2d35c0717cb6f81aea4e8a2ed47233c6"} Nov 02 14:23:11.490633 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:11.490529 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" podStartSLOduration=1.490508332 podStartE2EDuration="1.490508332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:23:11.486838375 +0000 UTC m=+1556.670685713" watchObservedRunningTime="2026-11-02 14:23:11.490508332 +0000 UTC m=+1556.674355670" Nov 02 14:23:11.843382 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:11.843300 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:12.600072 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:12.589261 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx"] Nov 02 14:23:12.600072 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:12.598274 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" Nov 02 14:23:12.615921 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:12.615890 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx"] Nov 02 14:23:12.623660 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:12.623443 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-944506-serving-cert\"" Nov 02 14:23:12.755843 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:12.755754 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d48de97-8d09-4a03-9f59-6422f88891a1-metrics-certs\") pod \"maas-api-e2e-extmod-a-944506-658755674f-7mghx\" (UID: \"8d48de97-8d09-4a03-9f59-6422f88891a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" Nov 02 14:23:12.756019 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:12.755912 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d48de97-8d09-4a03-9f59-6422f88891a1-maas-api-tls\") pod \"maas-api-e2e-extmod-a-944506-658755674f-7mghx\" (UID: \"8d48de97-8d09-4a03-9f59-6422f88891a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" Nov 02 14:23:12.756019 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:12.755938 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d48de97-8d09-4a03-9f59-6422f88891a1-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-944506-658755674f-7mghx\" (UID: \"8d48de97-8d09-4a03-9f59-6422f88891a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" Nov 02 14:23:12.756108 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:12.756092 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-545lt\" (UniqueName: \"kubernetes.io/projected/8d48de97-8d09-4a03-9f59-6422f88891a1-kube-api-access-545lt\") pod \"maas-api-e2e-extmod-a-944506-658755674f-7mghx\" (UID: \"8d48de97-8d09-4a03-9f59-6422f88891a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" Nov 02 14:23:12.852010 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:12.851975 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:12.857011 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:12.856976 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d48de97-8d09-4a03-9f59-6422f88891a1-maas-api-tls\") pod \"maas-api-e2e-extmod-a-944506-658755674f-7mghx\" (UID: \"8d48de97-8d09-4a03-9f59-6422f88891a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" Nov 02 14:23:12.857011 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:12.857025 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d48de97-8d09-4a03-9f59-6422f88891a1-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-944506-658755674f-7mghx\" (UID: \"8d48de97-8d09-4a03-9f59-6422f88891a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" Nov 02 14:23:12.857310 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:12.857103 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-545lt\" (UniqueName: \"kubernetes.io/projected/8d48de97-8d09-4a03-9f59-6422f88891a1-kube-api-access-545lt\") pod \"maas-api-e2e-extmod-a-944506-658755674f-7mghx\" (UID: \"8d48de97-8d09-4a03-9f59-6422f88891a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" Nov 02 14:23:12.857310 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:12.857186 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d48de97-8d09-4a03-9f59-6422f88891a1-metrics-certs\") pod \"maas-api-e2e-extmod-a-944506-658755674f-7mghx\" (UID: \"8d48de97-8d09-4a03-9f59-6422f88891a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" Nov 02 14:23:12.858007 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:12.857977 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d48de97-8d09-4a03-9f59-6422f88891a1-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-944506-658755674f-7mghx\" (UID: \"8d48de97-8d09-4a03-9f59-6422f88891a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" Nov 02 14:23:12.859939 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:12.859907 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d48de97-8d09-4a03-9f59-6422f88891a1-maas-api-tls\") pod \"maas-api-e2e-extmod-a-944506-658755674f-7mghx\" (UID: \"8d48de97-8d09-4a03-9f59-6422f88891a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" Nov 02 14:23:12.860084 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:12.859983 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d48de97-8d09-4a03-9f59-6422f88891a1-metrics-certs\") pod \"maas-api-e2e-extmod-a-944506-658755674f-7mghx\" (UID: \"8d48de97-8d09-4a03-9f59-6422f88891a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" Nov 02 14:23:12.885063 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:12.885034 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-545lt\" (UniqueName: \"kubernetes.io/projected/8d48de97-8d09-4a03-9f59-6422f88891a1-kube-api-access-545lt\") pod \"maas-api-e2e-extmod-a-944506-658755674f-7mghx\" (UID: \"8d48de97-8d09-4a03-9f59-6422f88891a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" Nov 02 14:23:12.933715 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:12.933677 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" Nov 02 14:23:13.243009 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:13.242973 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx"] Nov 02 14:23:13.244234 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:23:13.244192 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d48de97_8d09_4a03_9f59_6422f88891a1.slice/crio-e139187f3690a27c660235e2ece8369f9471a9797236c98abe75b049381c7350 WatchSource:0}: Error finding container e139187f3690a27c660235e2ece8369f9471a9797236c98abe75b049381c7350: Status 404 returned error can't find the container with id e139187f3690a27c660235e2ece8369f9471a9797236c98abe75b049381c7350 Nov 02 14:23:13.457196 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:13.457158 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" event={"ID":"8d48de97-8d09-4a03-9f59-6422f88891a1","Type":"ContainerStarted","Data":"e139187f3690a27c660235e2ece8369f9471a9797236c98abe75b049381c7350"} Nov 02 14:23:13.457459 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:13.457442 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:13.458399 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:13.458380 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:23:14.465837 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:14.465793 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" event={"ID":"8d48de97-8d09-4a03-9f59-6422f88891a1","Type":"ContainerStarted","Data":"7733af0f6aa359f8aaa79c8a3fc0fb3c25091ae6bf8b6fd106fafda3d3504962"} Nov 02 14:23:14.466290 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:14.465886 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" Nov 02 14:23:14.510645 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:14.510588 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" podStartSLOduration=2.009603861 podStartE2EDuration="2.510573981s" totalImagesPullingTime="500.97012ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:23:14.507660789 +0000 UTC m=+1559.691508127" watchObservedRunningTime="2026-11-02 14:23:14.510573981 +0000 UTC m=+1559.694421318" Nov 02 14:23:15.081572 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.081528 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8"] Nov 02 14:23:15.085759 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.085738 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8" Nov 02 14:23:15.102640 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.102613 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8416e2e2\"" Nov 02 14:23:15.102773 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.102695 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8416e2e2-dockercfg-49jbg\"" Nov 02 14:23:15.106236 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.106211 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8"] Nov 02 14:23:15.191483 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.187238 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng"] Nov 02 14:23:15.209970 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.192773 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7092854-cf7b-4b79-88e4-4747021cc8c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8\" (UID: \"a7092854-cf7b-4b79-88e4-4747021cc8c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8" Nov 02 14:23:15.234972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.210515 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ssdn\" (UniqueName: \"kubernetes.io/projected/a7092854-cf7b-4b79-88e4-4747021cc8c7-kube-api-access-7ssdn\") pod \"payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8\" (UID: \"a7092854-cf7b-4b79-88e4-4747021cc8c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8" Nov 02 14:23:15.263971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.262485 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng"] Nov 02 14:23:15.263971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.262605 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng" Nov 02 14:23:15.336649 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.336488 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7092854-cf7b-4b79-88e4-4747021cc8c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8\" (UID: \"a7092854-cf7b-4b79-88e4-4747021cc8c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8" Nov 02 14:23:15.336649 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.336587 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69120978-13b0-4cf6-8ead-5aeb9ab50376-plugins-config-volume\") pod \"payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng\" (UID: \"69120978-13b0-4cf6-8ead-5aeb9ab50376\") " pod="openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng" Nov 02 14:23:15.336852 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.336687 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvxhw\" (UniqueName: \"kubernetes.io/projected/69120978-13b0-4cf6-8ead-5aeb9ab50376-kube-api-access-nvxhw\") pod \"payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng\" (UID: \"69120978-13b0-4cf6-8ead-5aeb9ab50376\") " pod="openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng" Nov 02 14:23:15.336852 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.336720 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ssdn\" (UniqueName: \"kubernetes.io/projected/a7092854-cf7b-4b79-88e4-4747021cc8c7-kube-api-access-7ssdn\") pod \"payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8\" (UID: \"a7092854-cf7b-4b79-88e4-4747021cc8c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8" Nov 02 14:23:15.337724 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.337697 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7092854-cf7b-4b79-88e4-4747021cc8c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8\" (UID: \"a7092854-cf7b-4b79-88e4-4747021cc8c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8" Nov 02 14:23:15.360315 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.360287 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ssdn\" (UniqueName: \"kubernetes.io/projected/a7092854-cf7b-4b79-88e4-4747021cc8c7-kube-api-access-7ssdn\") pod \"payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8\" (UID: \"a7092854-cf7b-4b79-88e4-4747021cc8c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8" Nov 02 14:23:15.399769 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.399729 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8" Nov 02 14:23:15.438334 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.438300 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69120978-13b0-4cf6-8ead-5aeb9ab50376-plugins-config-volume\") pod \"payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng\" (UID: \"69120978-13b0-4cf6-8ead-5aeb9ab50376\") " pod="openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng" Nov 02 14:23:15.438496 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.438432 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvxhw\" (UniqueName: \"kubernetes.io/projected/69120978-13b0-4cf6-8ead-5aeb9ab50376-kube-api-access-nvxhw\") pod \"payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng\" (UID: \"69120978-13b0-4cf6-8ead-5aeb9ab50376\") " pod="openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng" Nov 02 14:23:15.439143 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.439118 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69120978-13b0-4cf6-8ead-5aeb9ab50376-plugins-config-volume\") pod \"payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng\" (UID: \"69120978-13b0-4cf6-8ead-5aeb9ab50376\") " pod="openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng" Nov 02 14:23:15.465829 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.465652 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvxhw\" (UniqueName: \"kubernetes.io/projected/69120978-13b0-4cf6-8ead-5aeb9ab50376-kube-api-access-nvxhw\") pod \"payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng\" (UID: \"69120978-13b0-4cf6-8ead-5aeb9ab50376\") " pod="openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng" Nov 02 14:23:15.611580 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.611486 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng" Nov 02 14:23:15.717444 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.717410 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8"] Nov 02 14:23:15.721981 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:23:15.721941 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7092854_cf7b_4b79_88e4_4747021cc8c7.slice/crio-679fdbfcd045bb628644b408e267eec69d9a3dce22464ad81dcdda288849e141 WatchSource:0}: Error finding container 679fdbfcd045bb628644b408e267eec69d9a3dce22464ad81dcdda288849e141: Status 404 returned error can't find the container with id 679fdbfcd045bb628644b408e267eec69d9a3dce22464ad81dcdda288849e141 Nov 02 14:23:15.923054 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:15.923028 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng"] Nov 02 14:23:15.924113 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:23:15.924088 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69120978_13b0_4cf6_8ead_5aeb9ab50376.slice/crio-3bc6ca530dd417d243a5047be6264004486c8427bf4c1905568cc4bbe9917f88 WatchSource:0}: Error finding container 3bc6ca530dd417d243a5047be6264004486c8427bf4c1905568cc4bbe9917f88: Status 404 returned error can't find the container with id 3bc6ca530dd417d243a5047be6264004486c8427bf4c1905568cc4bbe9917f88 Nov 02 14:23:16.491403 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:16.491365 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8" event={"ID":"a7092854-cf7b-4b79-88e4-4747021cc8c7","Type":"ContainerStarted","Data":"d98e1171920e7d762cb41961d8f38429714742834099133d0eba303a7a137b66"} Nov 02 14:23:16.491614 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:16.491445 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8" event={"ID":"a7092854-cf7b-4b79-88e4-4747021cc8c7","Type":"ContainerStarted","Data":"679fdbfcd045bb628644b408e267eec69d9a3dce22464ad81dcdda288849e141"} Nov 02 14:23:16.491708 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:16.491687 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8" Nov 02 14:23:16.497030 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:16.496996 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng" event={"ID":"69120978-13b0-4cf6-8ead-5aeb9ab50376","Type":"ContainerStarted","Data":"4e1013866bc6d807f6881a4bb47004be99d86d9025d6d8ec78a35fe21914722e"} Nov 02 14:23:16.497177 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:16.497052 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng" event={"ID":"69120978-13b0-4cf6-8ead-5aeb9ab50376","Type":"ContainerStarted","Data":"3bc6ca530dd417d243a5047be6264004486c8427bf4c1905568cc4bbe9917f88"} Nov 02 14:23:16.497177 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:16.497081 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng" Nov 02 14:23:16.532054 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:16.532003 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8" podStartSLOduration=1.531988994 podStartE2EDuration="1.531988994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:23:16.531403809 +0000 UTC m=+1561.715251147" watchObservedRunningTime="2026-11-02 14:23:16.531988994 +0000 UTC m=+1561.715836331" Nov 02 14:23:16.582851 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:16.582797 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng" podStartSLOduration=1.582782038 podStartE2EDuration="1.582782038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:23:16.578012921 +0000 UTC m=+1561.761860253" watchObservedRunningTime="2026-11-02 14:23:16.582782038 +0000 UTC m=+1561.766629374" Nov 02 14:23:17.414811 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:17.414785 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" Nov 02 14:23:20.488992 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:20.488919 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" Nov 02 14:23:27.507016 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:27.506919 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng" Nov 02 14:23:27.507498 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:27.507041 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8" Nov 02 14:23:36.011079 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:23:36.011042 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:23:58.631972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:58.617205 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-747f596d58-lncqr"] Nov 02 14:23:58.896158 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:58.896037 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" podUID="4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 02 14:23:58.915159 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:58.898302 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" podUID="d7037eab-3f5f-419b-94f6-784ff7331925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 02 14:23:58.916106 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:58.914536 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" podUID="df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:23:58.961971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:58.950743 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" podUID="0129df2e-9d21-45f7-9aa2-53777ce753b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:23:59.203969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:23:59.196103 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv" podUID="6d89a28a-7f87-4924-93c3-48ae7d11e56e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:24:03.617670 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:03.617537 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" podUID="3126935b-ac99-4227-b77d-b54707cd3dba" containerName="payload-processing" containerID="cri-o://97ec64359093c17c29ac99cecfb03394b53d1291d6ec205169e8e8d7d515294f" gracePeriod=25 Nov 02 14:24:03.954442 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:03.954415 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" Nov 02 14:24:03.972194 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:03.972165 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3126935b-ac99-4227-b77d-b54707cd3dba-kube-api-access-jn4fd\" (UniqueName: \"kubernetes.io/projected/3126935b-ac99-4227-b77d-b54707cd3dba-kube-api-access-jn4fd\") pod \"3126935b-ac99-4227-b77d-b54707cd3dba\" (UID: \"3126935b-ac99-4227-b77d-b54707cd3dba\") " Nov 02 14:24:03.972377 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:03.972289 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3126935b-ac99-4227-b77d-b54707cd3dba-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3126935b-ac99-4227-b77d-b54707cd3dba-routing-overlay-volume\") pod \"3126935b-ac99-4227-b77d-b54707cd3dba\" (UID: \"3126935b-ac99-4227-b77d-b54707cd3dba\") " Nov 02 14:24:03.972431 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:03.972402 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3126935b-ac99-4227-b77d-b54707cd3dba-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3126935b-ac99-4227-b77d-b54707cd3dba-provider-credentials\") pod \"3126935b-ac99-4227-b77d-b54707cd3dba\" (UID: \"3126935b-ac99-4227-b77d-b54707cd3dba\") " Nov 02 14:24:03.972479 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:03.972444 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3126935b-ac99-4227-b77d-b54707cd3dba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3126935b-ac99-4227-b77d-b54707cd3dba-plugins-config-volume\") pod \"3126935b-ac99-4227-b77d-b54707cd3dba\" (UID: \"3126935b-ac99-4227-b77d-b54707cd3dba\") " Nov 02 14:24:03.972671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:03.972644 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3126935b-ac99-4227-b77d-b54707cd3dba-routing-overlay-volume" pod "3126935b-ac99-4227-b77d-b54707cd3dba" (UID: "3126935b-ac99-4227-b77d-b54707cd3dba"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:24:03.972911 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:03.972882 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3126935b-ac99-4227-b77d-b54707cd3dba-plugins-config-volume" pod "3126935b-ac99-4227-b77d-b54707cd3dba" (UID: "3126935b-ac99-4227-b77d-b54707cd3dba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:24:03.973057 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:03.973021 4866 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3126935b-ac99-4227-b77d-b54707cd3dba-routing-overlay-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:03.973057 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:03.973040 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3126935b-ac99-4227-b77d-b54707cd3dba-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:03.974478 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:03.974449 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3126935b-ac99-4227-b77d-b54707cd3dba-kube-api-access-jn4fd" pod "3126935b-ac99-4227-b77d-b54707cd3dba" (UID: "3126935b-ac99-4227-b77d-b54707cd3dba"). InnerVolumeSpecName "kube-api-access-jn4fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:24:03.974603 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:03.974582 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3126935b-ac99-4227-b77d-b54707cd3dba-provider-credentials" pod "3126935b-ac99-4227-b77d-b54707cd3dba" (UID: "3126935b-ac99-4227-b77d-b54707cd3dba"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:24:04.073760 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:04.073721 4866 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3126935b-ac99-4227-b77d-b54707cd3dba-provider-credentials\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:04.073760 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:04.073752 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn4fd\" (UniqueName: \"kubernetes.io/projected/3126935b-ac99-4227-b77d-b54707cd3dba-kube-api-access-jn4fd\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:04.089346 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:04.089298 4866 generic.go:356] "Generic (PLEG): container finished" podID="3126935b-ac99-4227-b77d-b54707cd3dba" containerID="97ec64359093c17c29ac99cecfb03394b53d1291d6ec205169e8e8d7d515294f" exitCode=0 Nov 02 14:24:04.089551 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:04.089357 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" Nov 02 14:24:04.089551 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:04.089380 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" event={"ID":"3126935b-ac99-4227-b77d-b54707cd3dba","Type":"ContainerDied","Data":"97ec64359093c17c29ac99cecfb03394b53d1291d6ec205169e8e8d7d515294f"} Nov 02 14:24:04.089551 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:04.089441 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-747f596d58-lncqr" event={"ID":"3126935b-ac99-4227-b77d-b54707cd3dba","Type":"ContainerDied","Data":"c1bd48d4bac38b0dd5e6b49b21ca3f162fec7f77c3271a4f432bd100e5e2f551"} Nov 02 14:24:04.089551 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:04.089457 4866 scope.go:122] "RemoveContainer" containerID="97ec64359093c17c29ac99cecfb03394b53d1291d6ec205169e8e8d7d515294f" Nov 02 14:24:04.100335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:04.100315 4866 scope.go:122] "RemoveContainer" containerID="97ec64359093c17c29ac99cecfb03394b53d1291d6ec205169e8e8d7d515294f" Nov 02 14:24:04.100625 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:24:04.100602 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97ec64359093c17c29ac99cecfb03394b53d1291d6ec205169e8e8d7d515294f\": container with ID starting with 97ec64359093c17c29ac99cecfb03394b53d1291d6ec205169e8e8d7d515294f not found: ID does not exist" containerID="97ec64359093c17c29ac99cecfb03394b53d1291d6ec205169e8e8d7d515294f" Nov 02 14:24:04.100689 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:04.100633 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97ec64359093c17c29ac99cecfb03394b53d1291d6ec205169e8e8d7d515294f"} err="failed to get container status \"97ec64359093c17c29ac99cecfb03394b53d1291d6ec205169e8e8d7d515294f\": rpc error: code = NotFound desc = could not find container \"97ec64359093c17c29ac99cecfb03394b53d1291d6ec205169e8e8d7d515294f\": container with ID starting with 97ec64359093c17c29ac99cecfb03394b53d1291d6ec205169e8e8d7d515294f not found: ID does not exist" Nov 02 14:24:04.146123 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:04.146082 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-747f596d58-lncqr"] Nov 02 14:24:04.152131 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:04.152103 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-747f596d58-lncqr"] Nov 02 14:24:04.947473 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:04.947438 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3126935b-ac99-4227-b77d-b54707cd3dba" path="/var/lib/kubelet/pods/3126935b-ac99-4227-b77d-b54707cd3dba/volumes" Nov 02 14:24:06.011366 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:24:06.011328 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:24:08.125775 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.125735 4866 generic.go:356] "Generic (PLEG): container finished" podID="950d2085-58ef-47fb-a3f7-ef8cb5430c36" containerID="df13c949b4367e951fe1f6fc4d1d553f358e138f3bcb8b5be25fc087398a7d42" exitCode=0 Nov 02 14:24:08.126243 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.125798 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" event={"ID":"950d2085-58ef-47fb-a3f7-ef8cb5430c36","Type":"ContainerDied","Data":"df13c949b4367e951fe1f6fc4d1d553f358e138f3bcb8b5be25fc087398a7d42"} Nov 02 14:24:08.126243 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.126225 4866 scope.go:122] "RemoveContainer" containerID="df13c949b4367e951fe1f6fc4d1d553f358e138f3bcb8b5be25fc087398a7d42" Nov 02 14:24:08.127769 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.127741 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:24:08.127846 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.127810 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:24:08.127846 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.127843 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:24:08.422309 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.422220 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr"] Nov 02 14:24:08.422843 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.422829 4866 cpu_manager.go:414] "containerMap: removing container" podUID="3126935b-ac99-4227-b77d-b54707cd3dba" containerName="payload-processing" Nov 02 14:24:08.422891 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.422845 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3126935b-ac99-4227-b77d-b54707cd3dba" containerName="payload-processing" Nov 02 14:24:08.422956 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.422936 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="3126935b-ac99-4227-b77d-b54707cd3dba" containerName="payload-processing" Nov 02 14:24:08.425248 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.425231 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" Nov 02 14:24:08.440283 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.440232 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 14:24:08.440434 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.440413 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-56zdx\"" Nov 02 14:24:08.442840 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.442824 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1b98b233190d\"" Nov 02 14:24:08.450196 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.450166 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr"] Nov 02 14:24:08.537654 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.537615 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28800fff-0b1e-40aa-b1b1-9414258d76ed-routing-overlay-volume\") pod \"payload-processing-external-model-5c4dd54948-rb9gr\" (UID: \"28800fff-0b1e-40aa-b1b1-9414258d76ed\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" Nov 02 14:24:08.538003 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.537940 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28800fff-0b1e-40aa-b1b1-9414258d76ed-plugins-config-volume\") pod \"payload-processing-external-model-5c4dd54948-rb9gr\" (UID: \"28800fff-0b1e-40aa-b1b1-9414258d76ed\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" Nov 02 14:24:08.538087 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.538071 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/28800fff-0b1e-40aa-b1b1-9414258d76ed-provider-credentials\") pod \"payload-processing-external-model-5c4dd54948-rb9gr\" (UID: \"28800fff-0b1e-40aa-b1b1-9414258d76ed\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" Nov 02 14:24:08.538131 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.538118 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkx47\" (UniqueName: \"kubernetes.io/projected/28800fff-0b1e-40aa-b1b1-9414258d76ed-kube-api-access-pkx47\") pod \"payload-processing-external-model-5c4dd54948-rb9gr\" (UID: \"28800fff-0b1e-40aa-b1b1-9414258d76ed\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" Nov 02 14:24:08.541024 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.540998 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 14:24:08.541177 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.541023 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 14:24:08.541177 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.541026 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 14:24:08.639865 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.639821 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/28800fff-0b1e-40aa-b1b1-9414258d76ed-provider-credentials\") pod \"payload-processing-external-model-5c4dd54948-rb9gr\" (UID: \"28800fff-0b1e-40aa-b1b1-9414258d76ed\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" Nov 02 14:24:08.640069 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.639981 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkx47\" (UniqueName: \"kubernetes.io/projected/28800fff-0b1e-40aa-b1b1-9414258d76ed-kube-api-access-pkx47\") pod \"payload-processing-external-model-5c4dd54948-rb9gr\" (UID: \"28800fff-0b1e-40aa-b1b1-9414258d76ed\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" Nov 02 14:24:08.640069 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.640018 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28800fff-0b1e-40aa-b1b1-9414258d76ed-routing-overlay-volume\") pod \"payload-processing-external-model-5c4dd54948-rb9gr\" (UID: \"28800fff-0b1e-40aa-b1b1-9414258d76ed\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" Nov 02 14:24:08.640374 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.640352 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28800fff-0b1e-40aa-b1b1-9414258d76ed-plugins-config-volume\") pod \"payload-processing-external-model-5c4dd54948-rb9gr\" (UID: \"28800fff-0b1e-40aa-b1b1-9414258d76ed\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" Nov 02 14:24:08.640781 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.640757 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28800fff-0b1e-40aa-b1b1-9414258d76ed-routing-overlay-volume\") pod \"payload-processing-external-model-5c4dd54948-rb9gr\" (UID: \"28800fff-0b1e-40aa-b1b1-9414258d76ed\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" Nov 02 14:24:08.641096 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.641073 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28800fff-0b1e-40aa-b1b1-9414258d76ed-plugins-config-volume\") pod \"payload-processing-external-model-5c4dd54948-rb9gr\" (UID: \"28800fff-0b1e-40aa-b1b1-9414258d76ed\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" Nov 02 14:24:08.642611 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.642591 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/28800fff-0b1e-40aa-b1b1-9414258d76ed-provider-credentials\") pod \"payload-processing-external-model-5c4dd54948-rb9gr\" (UID: \"28800fff-0b1e-40aa-b1b1-9414258d76ed\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" Nov 02 14:24:08.678764 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.678674 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkx47\" (UniqueName: \"kubernetes.io/projected/28800fff-0b1e-40aa-b1b1-9414258d76ed-kube-api-access-pkx47\") pod \"payload-processing-external-model-5c4dd54948-rb9gr\" (UID: \"28800fff-0b1e-40aa-b1b1-9414258d76ed\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" Nov 02 14:24:08.739388 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:08.739353 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" Nov 02 14:24:09.058440 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:09.058395 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr"] Nov 02 14:24:09.067875 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:24:09.067835 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28800fff_0b1e_40aa_b1b1_9414258d76ed.slice/crio-e98d15abc53a9a6e368d3eae0f6b0d5f549e52532c28aadfc66223d9fb4bd69e WatchSource:0}: Error finding container e98d15abc53a9a6e368d3eae0f6b0d5f549e52532c28aadfc66223d9fb4bd69e: Status 404 returned error can't find the container with id e98d15abc53a9a6e368d3eae0f6b0d5f549e52532c28aadfc66223d9fb4bd69e Nov 02 14:24:09.136532 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:09.136497 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" event={"ID":"950d2085-58ef-47fb-a3f7-ef8cb5430c36","Type":"ContainerStarted","Data":"d71fbd502f902a74a04c60a9065c828e4e9ce1128803d27a46db9607ccba46a3"} Nov 02 14:24:09.138748 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:09.138716 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" event={"ID":"28800fff-0b1e-40aa-b1b1-9414258d76ed","Type":"ContainerStarted","Data":"e98d15abc53a9a6e368d3eae0f6b0d5f549e52532c28aadfc66223d9fb4bd69e"} Nov 02 14:24:09.843903 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:09.843868 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:24:09.845360 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:09.845335 4866 patch_prober.go:29] interesting pod/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 14:24:09.845477 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:09.845382 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" podUID="950d2085-58ef-47fb-a3f7-ef8cb5430c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 14:24:10.147139 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:10.147046 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" event={"ID":"28800fff-0b1e-40aa-b1b1-9414258d76ed","Type":"ContainerStarted","Data":"4cdac4f04ee337596f286b2d85fa8f4ac0d151cf686702e60bd2127250b14362"} Nov 02 14:24:10.190594 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:10.190538 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" podStartSLOduration=2.1905247 podStartE2EDuration="2.1905247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:24:10.188114887 +0000 UTC m=+1615.371962250" watchObservedRunningTime="2026-11-02 14:24:10.1905247 +0000 UTC m=+1615.374372037" Nov 02 14:24:10.844073 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:10.844037 4866 patch_prober.go:29] interesting pod/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 14:24:10.844263 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:10.844101 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" podUID="950d2085-58ef-47fb-a3f7-ef8cb5430c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 14:24:11.153108 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:11.153020 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" Nov 02 14:24:11.843092 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:11.843054 4866 patch_prober.go:29] interesting pod/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 14:24:11.843274 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:11.843127 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" podUID="950d2085-58ef-47fb-a3f7-ef8cb5430c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 14:24:12.844870 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:12.844365 4866 patch_prober.go:29] interesting pod/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 14:24:12.844870 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:12.844438 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" podUID="950d2085-58ef-47fb-a3f7-ef8cb5430c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 14:24:13.457866 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:13.457829 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:24:13.844149 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:13.844065 4866 patch_prober.go:29] interesting pod/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 14:24:13.844149 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:13.844133 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" podUID="950d2085-58ef-47fb-a3f7-ef8cb5430c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 14:24:14.862763 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:14.862725 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:24:14.864480 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:14.864452 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:24:15.812067 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:15.812030 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng"] Nov 02 14:24:15.852529 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:15.852490 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8"] Nov 02 14:24:18.967993 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:18.967889 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-8q5hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:24:18.968876 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:18.968531 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" podUID="1fbf701c-a1aa-4031-8112-4e69e00e103b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:24:20.812691 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:20.812649 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng" podUID="69120978-13b0-4cf6-8ead-5aeb9ab50376" containerName="payload-processing" containerID="cri-o://4e1013866bc6d807f6881a4bb47004be99d86d9025d6d8ec78a35fe21914722e" gracePeriod=25 Nov 02 14:24:20.852784 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:20.852744 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8" podUID="a7092854-cf7b-4b79-88e4-4747021cc8c7" containerName="payload-pre-processing" containerID="cri-o://d98e1171920e7d762cb41961d8f38429714742834099133d0eba303a7a137b66" gracePeriod=25 Nov 02 14:24:21.114414 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.114385 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng" Nov 02 14:24:21.158864 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.158835 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8" Nov 02 14:24:21.207700 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.207663 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7092854-cf7b-4b79-88e4-4747021cc8c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7092854-cf7b-4b79-88e4-4747021cc8c7-plugins-config-volume\") pod \"a7092854-cf7b-4b79-88e4-4747021cc8c7\" (UID: \"a7092854-cf7b-4b79-88e4-4747021cc8c7\") " Nov 02 14:24:21.207857 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.207825 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7092854-cf7b-4b79-88e4-4747021cc8c7-kube-api-access-7ssdn\" (UniqueName: \"kubernetes.io/projected/a7092854-cf7b-4b79-88e4-4747021cc8c7-kube-api-access-7ssdn\") pod \"a7092854-cf7b-4b79-88e4-4747021cc8c7\" (UID: \"a7092854-cf7b-4b79-88e4-4747021cc8c7\") " Nov 02 14:24:21.207942 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.207894 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69120978-13b0-4cf6-8ead-5aeb9ab50376-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69120978-13b0-4cf6-8ead-5aeb9ab50376-plugins-config-volume\") pod \"69120978-13b0-4cf6-8ead-5aeb9ab50376\" (UID: \"69120978-13b0-4cf6-8ead-5aeb9ab50376\") " Nov 02 14:24:21.208031 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.208015 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69120978-13b0-4cf6-8ead-5aeb9ab50376-kube-api-access-nvxhw\" (UniqueName: \"kubernetes.io/projected/69120978-13b0-4cf6-8ead-5aeb9ab50376-kube-api-access-nvxhw\") pod \"69120978-13b0-4cf6-8ead-5aeb9ab50376\" (UID: \"69120978-13b0-4cf6-8ead-5aeb9ab50376\") " Nov 02 14:24:21.208325 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.208126 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7092854-cf7b-4b79-88e4-4747021cc8c7-plugins-config-volume" pod "a7092854-cf7b-4b79-88e4-4747021cc8c7" (UID: "a7092854-cf7b-4b79-88e4-4747021cc8c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:24:21.208743 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.208537 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69120978-13b0-4cf6-8ead-5aeb9ab50376-plugins-config-volume" pod "69120978-13b0-4cf6-8ead-5aeb9ab50376" (UID: "69120978-13b0-4cf6-8ead-5aeb9ab50376"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:24:21.208743 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.208700 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7092854-cf7b-4b79-88e4-4747021cc8c7-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:21.208743 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.208722 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69120978-13b0-4cf6-8ead-5aeb9ab50376-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:21.210995 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.210967 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7092854-cf7b-4b79-88e4-4747021cc8c7-kube-api-access-7ssdn" pod "a7092854-cf7b-4b79-88e4-4747021cc8c7" (UID: "a7092854-cf7b-4b79-88e4-4747021cc8c7"). InnerVolumeSpecName "kube-api-access-7ssdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:24:21.211112 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.211025 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69120978-13b0-4cf6-8ead-5aeb9ab50376-kube-api-access-nvxhw" pod "69120978-13b0-4cf6-8ead-5aeb9ab50376" (UID: "69120978-13b0-4cf6-8ead-5aeb9ab50376"). InnerVolumeSpecName "kube-api-access-nvxhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:24:21.252830 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.252796 4866 generic.go:356] "Generic (PLEG): container finished" podID="a7092854-cf7b-4b79-88e4-4747021cc8c7" containerID="d98e1171920e7d762cb41961d8f38429714742834099133d0eba303a7a137b66" exitCode=0 Nov 02 14:24:21.253021 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.252847 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8" Nov 02 14:24:21.253021 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.252884 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8" event={"ID":"a7092854-cf7b-4b79-88e4-4747021cc8c7","Type":"ContainerDied","Data":"d98e1171920e7d762cb41961d8f38429714742834099133d0eba303a7a137b66"} Nov 02 14:24:21.253021 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.252968 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8" event={"ID":"a7092854-cf7b-4b79-88e4-4747021cc8c7","Type":"ContainerDied","Data":"679fdbfcd045bb628644b408e267eec69d9a3dce22464ad81dcdda288849e141"} Nov 02 14:24:21.253021 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.253001 4866 scope.go:122] "RemoveContainer" containerID="d98e1171920e7d762cb41961d8f38429714742834099133d0eba303a7a137b66" Nov 02 14:24:21.254519 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.254496 4866 generic.go:356] "Generic (PLEG): container finished" podID="69120978-13b0-4cf6-8ead-5aeb9ab50376" containerID="4e1013866bc6d807f6881a4bb47004be99d86d9025d6d8ec78a35fe21914722e" exitCode=0 Nov 02 14:24:21.254642 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.254541 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng" Nov 02 14:24:21.254642 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.254580 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng" event={"ID":"69120978-13b0-4cf6-8ead-5aeb9ab50376","Type":"ContainerDied","Data":"4e1013866bc6d807f6881a4bb47004be99d86d9025d6d8ec78a35fe21914722e"} Nov 02 14:24:21.254642 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.254632 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng" event={"ID":"69120978-13b0-4cf6-8ead-5aeb9ab50376","Type":"ContainerDied","Data":"3bc6ca530dd417d243a5047be6264004486c8427bf4c1905568cc4bbe9917f88"} Nov 02 14:24:21.262837 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.262821 4866 scope.go:122] "RemoveContainer" containerID="d98e1171920e7d762cb41961d8f38429714742834099133d0eba303a7a137b66" Nov 02 14:24:21.263159 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:24:21.263136 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d98e1171920e7d762cb41961d8f38429714742834099133d0eba303a7a137b66\": container with ID starting with d98e1171920e7d762cb41961d8f38429714742834099133d0eba303a7a137b66 not found: ID does not exist" containerID="d98e1171920e7d762cb41961d8f38429714742834099133d0eba303a7a137b66" Nov 02 14:24:21.263252 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.263170 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d98e1171920e7d762cb41961d8f38429714742834099133d0eba303a7a137b66"} err="failed to get container status \"d98e1171920e7d762cb41961d8f38429714742834099133d0eba303a7a137b66\": rpc error: code = NotFound desc = could not find container \"d98e1171920e7d762cb41961d8f38429714742834099133d0eba303a7a137b66\": container with ID starting with d98e1171920e7d762cb41961d8f38429714742834099133d0eba303a7a137b66 not found: ID does not exist" Nov 02 14:24:21.263252 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.263189 4866 scope.go:122] "RemoveContainer" containerID="4e1013866bc6d807f6881a4bb47004be99d86d9025d6d8ec78a35fe21914722e" Nov 02 14:24:21.272575 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.272550 4866 scope.go:122] "RemoveContainer" containerID="4e1013866bc6d807f6881a4bb47004be99d86d9025d6d8ec78a35fe21914722e" Nov 02 14:24:21.272897 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:24:21.272874 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e1013866bc6d807f6881a4bb47004be99d86d9025d6d8ec78a35fe21914722e\": container with ID starting with 4e1013866bc6d807f6881a4bb47004be99d86d9025d6d8ec78a35fe21914722e not found: ID does not exist" containerID="4e1013866bc6d807f6881a4bb47004be99d86d9025d6d8ec78a35fe21914722e" Nov 02 14:24:21.272956 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.272907 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e1013866bc6d807f6881a4bb47004be99d86d9025d6d8ec78a35fe21914722e"} err="failed to get container status \"4e1013866bc6d807f6881a4bb47004be99d86d9025d6d8ec78a35fe21914722e\": rpc error: code = NotFound desc = could not find container \"4e1013866bc6d807f6881a4bb47004be99d86d9025d6d8ec78a35fe21914722e\": container with ID starting with 4e1013866bc6d807f6881a4bb47004be99d86d9025d6d8ec78a35fe21914722e not found: ID does not exist" Nov 02 14:24:21.308465 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.308431 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng"] Nov 02 14:24:21.309787 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.309766 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ssdn\" (UniqueName: \"kubernetes.io/projected/a7092854-cf7b-4b79-88e4-4747021cc8c7-kube-api-access-7ssdn\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:21.309787 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.309788 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvxhw\" (UniqueName: \"kubernetes.io/projected/69120978-13b0-4cf6-8ead-5aeb9ab50376-kube-api-access-nvxhw\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:21.312801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.312770 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8416e2e2-75fd84c66f-fxbng"] Nov 02 14:24:21.359166 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.359054 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8"] Nov 02 14:24:21.362713 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:21.362679 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8416e2e2-6d96d9fd4c-lf5b8"] Nov 02 14:24:22.162964 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:22.162866 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" Nov 02 14:24:22.947074 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:22.947038 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69120978-13b0-4cf6-8ead-5aeb9ab50376" path="/var/lib/kubelet/pods/69120978-13b0-4cf6-8ead-5aeb9ab50376/volumes" Nov 02 14:24:22.947494 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:22.947479 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7092854-cf7b-4b79-88e4-4747021cc8c7" path="/var/lib/kubelet/pods/a7092854-cf7b-4b79-88e4-4747021cc8c7/volumes" Nov 02 14:24:30.360104 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:30.360068 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj"] Nov 02 14:24:30.361065 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:30.360373 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" podUID="950d2085-58ef-47fb-a3f7-ef8cb5430c36" containerName="istio-proxy" containerID="cri-o://d71fbd502f902a74a04c60a9065c828e4e9ce1128803d27a46db9607ccba46a3" gracePeriod=30 Nov 02 14:24:35.812681 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:35.812652 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:24:35.936467 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:35.936373 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-token\" (UniqueName: \"kubernetes.io/projected/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-token\") pod \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " Nov 02 14:24:35.937015 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:35.936983 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-credential-socket\") pod \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " Nov 02 14:24:35.937147 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:35.937042 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-workload-socket\") pod \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " Nov 02 14:24:35.937210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:35.937173 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-data\") pod \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " Nov 02 14:24:35.937268 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:35.937244 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istiod-ca-cert\") pod \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " Nov 02 14:24:35.937330 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:35.937297 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950d2085-58ef-47fb-a3f7-ef8cb5430c36-kube-api-access-7k8tj\" (UniqueName: \"kubernetes.io/projected/950d2085-58ef-47fb-a3f7-ef8cb5430c36-kube-api-access-7k8tj\") pod \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " Nov 02 14:24:35.937385 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:35.937332 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-envoy\") pod \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " Nov 02 14:24:35.937385 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:35.937361 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-podinfo\") pod \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " Nov 02 14:24:35.937488 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:35.937404 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-workload-certs\") pod \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\" (UID: \"950d2085-58ef-47fb-a3f7-ef8cb5430c36\") " Nov 02 14:24:35.941931 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:35.941895 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-token" pod "950d2085-58ef-47fb-a3f7-ef8cb5430c36" (UID: "950d2085-58ef-47fb-a3f7-ef8cb5430c36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:24:35.942184 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:35.942161 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-workload-certs" pod "950d2085-58ef-47fb-a3f7-ef8cb5430c36" (UID: "950d2085-58ef-47fb-a3f7-ef8cb5430c36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:35.945169 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:35.945142 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-workload-socket" pod "950d2085-58ef-47fb-a3f7-ef8cb5430c36" (UID: "950d2085-58ef-47fb-a3f7-ef8cb5430c36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:35.945326 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:35.945164 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-data" pod "950d2085-58ef-47fb-a3f7-ef8cb5430c36" (UID: "950d2085-58ef-47fb-a3f7-ef8cb5430c36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:35.945414 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:35.945373 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istiod-ca-cert" pod "950d2085-58ef-47fb-a3f7-ef8cb5430c36" (UID: "950d2085-58ef-47fb-a3f7-ef8cb5430c36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:24:35.945861 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:35.945835 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950d2085-58ef-47fb-a3f7-ef8cb5430c36-kube-api-access-7k8tj" pod "950d2085-58ef-47fb-a3f7-ef8cb5430c36" (UID: "950d2085-58ef-47fb-a3f7-ef8cb5430c36"). InnerVolumeSpecName "kube-api-access-7k8tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:24:35.945918 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:35.945856 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-podinfo" pod "950d2085-58ef-47fb-a3f7-ef8cb5430c36" (UID: "950d2085-58ef-47fb-a3f7-ef8cb5430c36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:24:35.946308 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:35.946285 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-envoy" pod "950d2085-58ef-47fb-a3f7-ef8cb5430c36" (UID: "950d2085-58ef-47fb-a3f7-ef8cb5430c36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:35.950611 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:35.950571 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-credential-socket" pod "950d2085-58ef-47fb-a3f7-ef8cb5430c36" (UID: "950d2085-58ef-47fb-a3f7-ef8cb5430c36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:36.011617 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:24:36.011583 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:24:36.039813 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.039766 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:36.039813 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.039803 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:36.039813 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.039819 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:36.040088 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.039832 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:36.040088 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.039848 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k8tj\" (UniqueName: \"kubernetes.io/projected/950d2085-58ef-47fb-a3f7-ef8cb5430c36-kube-api-access-7k8tj\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:36.040088 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.039885 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:36.040088 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.039909 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:36.040088 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.039922 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950d2085-58ef-47fb-a3f7-ef8cb5430c36-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:36.040088 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.039936 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950d2085-58ef-47fb-a3f7-ef8cb5430c36-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:36.408881 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.408844 4866 generic.go:356] "Generic (PLEG): container finished" podID="950d2085-58ef-47fb-a3f7-ef8cb5430c36" containerID="d71fbd502f902a74a04c60a9065c828e4e9ce1128803d27a46db9607ccba46a3" exitCode=0 Nov 02 14:24:36.409072 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.408891 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" Nov 02 14:24:36.409072 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.408923 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" event={"ID":"950d2085-58ef-47fb-a3f7-ef8cb5430c36","Type":"ContainerDied","Data":"d71fbd502f902a74a04c60a9065c828e4e9ce1128803d27a46db9607ccba46a3"} Nov 02 14:24:36.409072 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.408998 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj" event={"ID":"950d2085-58ef-47fb-a3f7-ef8cb5430c36","Type":"ContainerDied","Data":"2a34ab698800ba2888e5603eb05e4ffa2d35c0717cb6f81aea4e8a2ed47233c6"} Nov 02 14:24:36.409072 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.409034 4866 scope.go:122] "RemoveContainer" containerID="d71fbd502f902a74a04c60a9065c828e4e9ce1128803d27a46db9607ccba46a3" Nov 02 14:24:36.421129 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.421108 4866 scope.go:122] "RemoveContainer" containerID="df13c949b4367e951fe1f6fc4d1d553f358e138f3bcb8b5be25fc087398a7d42" Nov 02 14:24:36.430538 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.430518 4866 scope.go:122] "RemoveContainer" containerID="d71fbd502f902a74a04c60a9065c828e4e9ce1128803d27a46db9607ccba46a3" Nov 02 14:24:36.430829 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:24:36.430812 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d71fbd502f902a74a04c60a9065c828e4e9ce1128803d27a46db9607ccba46a3\": container with ID starting with d71fbd502f902a74a04c60a9065c828e4e9ce1128803d27a46db9607ccba46a3 not found: ID does not exist" containerID="d71fbd502f902a74a04c60a9065c828e4e9ce1128803d27a46db9607ccba46a3" Nov 02 14:24:36.430914 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.430835 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71fbd502f902a74a04c60a9065c828e4e9ce1128803d27a46db9607ccba46a3"} err="failed to get container status \"d71fbd502f902a74a04c60a9065c828e4e9ce1128803d27a46db9607ccba46a3\": rpc error: code = NotFound desc = could not find container \"d71fbd502f902a74a04c60a9065c828e4e9ce1128803d27a46db9607ccba46a3\": container with ID starting with d71fbd502f902a74a04c60a9065c828e4e9ce1128803d27a46db9607ccba46a3 not found: ID does not exist" Nov 02 14:24:36.430914 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.430850 4866 scope.go:122] "RemoveContainer" containerID="df13c949b4367e951fe1f6fc4d1d553f358e138f3bcb8b5be25fc087398a7d42" Nov 02 14:24:36.431169 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:24:36.431142 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df13c949b4367e951fe1f6fc4d1d553f358e138f3bcb8b5be25fc087398a7d42\": container with ID starting with df13c949b4367e951fe1f6fc4d1d553f358e138f3bcb8b5be25fc087398a7d42 not found: ID does not exist" containerID="df13c949b4367e951fe1f6fc4d1d553f358e138f3bcb8b5be25fc087398a7d42" Nov 02 14:24:36.431217 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.431177 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df13c949b4367e951fe1f6fc4d1d553f358e138f3bcb8b5be25fc087398a7d42"} err="failed to get container status \"df13c949b4367e951fe1f6fc4d1d553f358e138f3bcb8b5be25fc087398a7d42\": rpc error: code = NotFound desc = could not find container \"df13c949b4367e951fe1f6fc4d1d553f358e138f3bcb8b5be25fc087398a7d42\": container with ID starting with df13c949b4367e951fe1f6fc4d1d553f358e138f3bcb8b5be25fc087398a7d42 not found: ID does not exist" Nov 02 14:24:36.472656 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.472620 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj"] Nov 02 14:24:36.492778 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.492742 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8416e2e2-openshift-default-677756d6f9-t52kj"] Nov 02 14:24:36.691125 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.691086 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-778c654c9-lrkxv"] Nov 02 14:24:36.696301 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.692556 4866 cpu_manager.go:414] "containerMap: removing container" podUID="950d2085-58ef-47fb-a3f7-ef8cb5430c36" containerName="istio-proxy" Nov 02 14:24:36.696301 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.692582 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="950d2085-58ef-47fb-a3f7-ef8cb5430c36" containerName="istio-proxy" Nov 02 14:24:36.696301 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.692604 4866 cpu_manager.go:414] "containerMap: removing container" podUID="69120978-13b0-4cf6-8ead-5aeb9ab50376" containerName="payload-processing" Nov 02 14:24:36.696301 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.692611 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69120978-13b0-4cf6-8ead-5aeb9ab50376" containerName="payload-processing" Nov 02 14:24:36.696301 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.692682 4866 cpu_manager.go:414] "containerMap: removing container" podUID="950d2085-58ef-47fb-a3f7-ef8cb5430c36" containerName="istio-proxy" Nov 02 14:24:36.696301 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.692691 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="950d2085-58ef-47fb-a3f7-ef8cb5430c36" containerName="istio-proxy" Nov 02 14:24:36.696301 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.692747 4866 cpu_manager.go:414] "containerMap: removing container" podUID="a7092854-cf7b-4b79-88e4-4747021cc8c7" containerName="payload-pre-processing" Nov 02 14:24:36.696301 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.692761 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7092854-cf7b-4b79-88e4-4747021cc8c7" containerName="payload-pre-processing" Nov 02 14:24:36.696301 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.693032 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7092854-cf7b-4b79-88e4-4747021cc8c7" containerName="payload-pre-processing" Nov 02 14:24:36.696301 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.693054 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="950d2085-58ef-47fb-a3f7-ef8cb5430c36" containerName="istio-proxy" Nov 02 14:24:36.696301 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.693071 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="950d2085-58ef-47fb-a3f7-ef8cb5430c36" containerName="istio-proxy" Nov 02 14:24:36.696301 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.693082 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="69120978-13b0-4cf6-8ead-5aeb9ab50376" containerName="payload-processing" Nov 02 14:24:36.705898 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.705859 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" Nov 02 14:24:36.717310 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.717276 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-778c654c9-lrkxv"] Nov 02 14:24:36.720892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.720863 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1dd67d8b74fd\"" Nov 02 14:24:36.721065 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.721031 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-eaf1e3aae04d\"" Nov 02 14:24:36.849622 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.849593 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0582afd-319e-4051-bc1e-b7470824044c-provider-credentials\") pod \"payload-processing-external-model-778c654c9-lrkxv\" (UID: \"a0582afd-319e-4051-bc1e-b7470824044c\") " pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" Nov 02 14:24:36.850137 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.849644 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0582afd-319e-4051-bc1e-b7470824044c-routing-overlay-volume\") pod \"payload-processing-external-model-778c654c9-lrkxv\" (UID: \"a0582afd-319e-4051-bc1e-b7470824044c\") " pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" Nov 02 14:24:36.850137 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.849867 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0582afd-319e-4051-bc1e-b7470824044c-plugins-config-volume\") pod \"payload-processing-external-model-778c654c9-lrkxv\" (UID: \"a0582afd-319e-4051-bc1e-b7470824044c\") " pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" Nov 02 14:24:36.850137 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.850011 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f759v\" (UniqueName: \"kubernetes.io/projected/a0582afd-319e-4051-bc1e-b7470824044c-kube-api-access-f759v\") pod \"payload-processing-external-model-778c654c9-lrkxv\" (UID: \"a0582afd-319e-4051-bc1e-b7470824044c\") " pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" Nov 02 14:24:36.946183 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.946092 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="950d2085-58ef-47fb-a3f7-ef8cb5430c36" path="/var/lib/kubelet/pods/950d2085-58ef-47fb-a3f7-ef8cb5430c36/volumes" Nov 02 14:24:36.951314 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.951289 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f759v\" (UniqueName: \"kubernetes.io/projected/a0582afd-319e-4051-bc1e-b7470824044c-kube-api-access-f759v\") pod \"payload-processing-external-model-778c654c9-lrkxv\" (UID: \"a0582afd-319e-4051-bc1e-b7470824044c\") " pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" Nov 02 14:24:36.951483 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.951357 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0582afd-319e-4051-bc1e-b7470824044c-provider-credentials\") pod \"payload-processing-external-model-778c654c9-lrkxv\" (UID: \"a0582afd-319e-4051-bc1e-b7470824044c\") " pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" Nov 02 14:24:36.951483 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.951392 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0582afd-319e-4051-bc1e-b7470824044c-routing-overlay-volume\") pod \"payload-processing-external-model-778c654c9-lrkxv\" (UID: \"a0582afd-319e-4051-bc1e-b7470824044c\") " pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" Nov 02 14:24:36.951609 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.951487 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0582afd-319e-4051-bc1e-b7470824044c-plugins-config-volume\") pod \"payload-processing-external-model-778c654c9-lrkxv\" (UID: \"a0582afd-319e-4051-bc1e-b7470824044c\") " pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" Nov 02 14:24:36.952203 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.952176 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0582afd-319e-4051-bc1e-b7470824044c-routing-overlay-volume\") pod \"payload-processing-external-model-778c654c9-lrkxv\" (UID: \"a0582afd-319e-4051-bc1e-b7470824044c\") " pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" Nov 02 14:24:36.952317 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.952291 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0582afd-319e-4051-bc1e-b7470824044c-plugins-config-volume\") pod \"payload-processing-external-model-778c654c9-lrkxv\" (UID: \"a0582afd-319e-4051-bc1e-b7470824044c\") " pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" Nov 02 14:24:36.954075 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.954054 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0582afd-319e-4051-bc1e-b7470824044c-provider-credentials\") pod \"payload-processing-external-model-778c654c9-lrkxv\" (UID: \"a0582afd-319e-4051-bc1e-b7470824044c\") " pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" Nov 02 14:24:36.977061 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:36.977031 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f759v\" (UniqueName: \"kubernetes.io/projected/a0582afd-319e-4051-bc1e-b7470824044c-kube-api-access-f759v\") pod \"payload-processing-external-model-778c654c9-lrkxv\" (UID: \"a0582afd-319e-4051-bc1e-b7470824044c\") " pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" Nov 02 14:24:37.024310 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:37.024278 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" Nov 02 14:24:37.327589 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:37.327557 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-778c654c9-lrkxv"] Nov 02 14:24:37.328106 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:24:37.328077 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0582afd_319e_4051_bc1e_b7470824044c.slice/crio-ee5b6125a9f92009bb06d957902d07fed140364060654739327ac79fe0960d27 WatchSource:0}: Error finding container ee5b6125a9f92009bb06d957902d07fed140364060654739327ac79fe0960d27: Status 404 returned error can't find the container with id ee5b6125a9f92009bb06d957902d07fed140364060654739327ac79fe0960d27 Nov 02 14:24:37.416639 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:37.416605 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" event={"ID":"a0582afd-319e-4051-bc1e-b7470824044c","Type":"ContainerStarted","Data":"2c2aa2110badf99b8811c6f41f204943d9f52bf8db5a87296192135c981bca41"} Nov 02 14:24:37.416806 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:37.416667 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" event={"ID":"a0582afd-319e-4051-bc1e-b7470824044c","Type":"ContainerStarted","Data":"ee5b6125a9f92009bb06d957902d07fed140364060654739327ac79fe0960d27"} Nov 02 14:24:37.416806 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:37.416694 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" Nov 02 14:24:37.458480 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:37.458431 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" podStartSLOduration=1.458417668 podStartE2EDuration="1.458417668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:24:37.457380773 +0000 UTC m=+1642.641228106" watchObservedRunningTime="2026-11-02 14:24:37.458417668 +0000 UTC m=+1642.642265004" Nov 02 14:24:37.813048 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:37.813010 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f"] Nov 02 14:24:37.817068 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:37.817051 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f" Nov 02 14:24:37.838295 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:37.838260 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b3690233\"" Nov 02 14:24:37.838428 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:37.838388 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b3690233-dockercfg-h2lrd\"" Nov 02 14:24:37.838505 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:37.838442 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f"] Nov 02 14:24:38.056042 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:38.055918 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h"] Nov 02 14:24:38.062995 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:38.062330 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h" Nov 02 14:24:38.065757 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:38.065690 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h"] Nov 02 14:24:38.066356 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:38.066330 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2rmr\" (UniqueName: \"kubernetes.io/projected/3ef4069e-ce7d-41e9-9493-2ea93ddbdefa-kube-api-access-n2rmr\") pod \"payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f\" (UID: \"3ef4069e-ce7d-41e9-9493-2ea93ddbdefa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f" Nov 02 14:24:38.066477 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:38.066370 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef4069e-ce7d-41e9-9493-2ea93ddbdefa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f\" (UID: \"3ef4069e-ce7d-41e9-9493-2ea93ddbdefa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f" Nov 02 14:24:38.167639 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:38.167606 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvjzg\" (UniqueName: \"kubernetes.io/projected/c83de3cb-ad5f-4e57-83e4-d16913ab66e3-kube-api-access-vvjzg\") pod \"payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h\" (UID: \"c83de3cb-ad5f-4e57-83e4-d16913ab66e3\") " pod="openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h" Nov 02 14:24:38.167815 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:38.167728 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2rmr\" (UniqueName: \"kubernetes.io/projected/3ef4069e-ce7d-41e9-9493-2ea93ddbdefa-kube-api-access-n2rmr\") pod \"payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f\" (UID: \"3ef4069e-ce7d-41e9-9493-2ea93ddbdefa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f" Nov 02 14:24:38.167815 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:38.167786 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef4069e-ce7d-41e9-9493-2ea93ddbdefa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f\" (UID: \"3ef4069e-ce7d-41e9-9493-2ea93ddbdefa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f" Nov 02 14:24:38.168012 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:38.167885 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83de3cb-ad5f-4e57-83e4-d16913ab66e3-plugins-config-volume\") pod \"payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h\" (UID: \"c83de3cb-ad5f-4e57-83e4-d16913ab66e3\") " pod="openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h" Nov 02 14:24:38.168505 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:38.168487 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef4069e-ce7d-41e9-9493-2ea93ddbdefa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f\" (UID: \"3ef4069e-ce7d-41e9-9493-2ea93ddbdefa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f" Nov 02 14:24:38.191404 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:38.191366 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2rmr\" (UniqueName: \"kubernetes.io/projected/3ef4069e-ce7d-41e9-9493-2ea93ddbdefa-kube-api-access-n2rmr\") pod \"payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f\" (UID: \"3ef4069e-ce7d-41e9-9493-2ea93ddbdefa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f" Nov 02 14:24:38.269205 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:38.269166 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvjzg\" (UniqueName: \"kubernetes.io/projected/c83de3cb-ad5f-4e57-83e4-d16913ab66e3-kube-api-access-vvjzg\") pod \"payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h\" (UID: \"c83de3cb-ad5f-4e57-83e4-d16913ab66e3\") " pod="openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h" Nov 02 14:24:38.269383 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:38.269296 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83de3cb-ad5f-4e57-83e4-d16913ab66e3-plugins-config-volume\") pod \"payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h\" (UID: \"c83de3cb-ad5f-4e57-83e4-d16913ab66e3\") " pod="openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h" Nov 02 14:24:38.270037 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:38.270019 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83de3cb-ad5f-4e57-83e4-d16913ab66e3-plugins-config-volume\") pod \"payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h\" (UID: \"c83de3cb-ad5f-4e57-83e4-d16913ab66e3\") " pod="openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h" Nov 02 14:24:38.293390 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:38.293360 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvjzg\" (UniqueName: \"kubernetes.io/projected/c83de3cb-ad5f-4e57-83e4-d16913ab66e3-kube-api-access-vvjzg\") pod \"payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h\" (UID: \"c83de3cb-ad5f-4e57-83e4-d16913ab66e3\") " pod="openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h" Nov 02 14:24:38.379390 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:38.379307 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h" Nov 02 14:24:38.431317 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:38.431285 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f" Nov 02 14:24:38.714392 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:38.714355 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h"] Nov 02 14:24:38.779014 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:38.778985 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f"] Nov 02 14:24:38.780669 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:24:38.780638 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef4069e_ce7d_41e9_9493_2ea93ddbdefa.slice/crio-e3a41b03d8de420b4eb9b56c66b80cc5102a24760f8c4f0fa9b77f4d22ae7d60 WatchSource:0}: Error finding container e3a41b03d8de420b4eb9b56c66b80cc5102a24760f8c4f0fa9b77f4d22ae7d60: Status 404 returned error can't find the container with id e3a41b03d8de420b4eb9b56c66b80cc5102a24760f8c4f0fa9b77f4d22ae7d60 Nov 02 14:24:39.434532 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:39.434482 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h" event={"ID":"c83de3cb-ad5f-4e57-83e4-d16913ab66e3","Type":"ContainerStarted","Data":"3fa00b6d0acb50dcd3f8a97be8946e14701e5d2b7254d7b22549f0c592279f6a"} Nov 02 14:24:39.435060 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:39.434550 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h" event={"ID":"c83de3cb-ad5f-4e57-83e4-d16913ab66e3","Type":"ContainerStarted","Data":"90fbd2b7d7cb46f226d9e56964d3a55256d9be78d089e5d94e6c608f927f8390"} Nov 02 14:24:39.435060 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:39.434591 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h" Nov 02 14:24:39.436366 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:39.436338 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f" event={"ID":"3ef4069e-ce7d-41e9-9493-2ea93ddbdefa","Type":"ContainerStarted","Data":"2bf4f88f3653d2a68fcb5eaf22de84c7e71d7ea20c163a32c205a93a9c6564b2"} Nov 02 14:24:39.436497 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:39.436382 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f" event={"ID":"3ef4069e-ce7d-41e9-9493-2ea93ddbdefa","Type":"ContainerStarted","Data":"e3a41b03d8de420b4eb9b56c66b80cc5102a24760f8c4f0fa9b77f4d22ae7d60"} Nov 02 14:24:39.436497 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:39.436413 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f" Nov 02 14:24:39.474515 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:39.474464 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h" podStartSLOduration=2.474447198 podStartE2EDuration="2.474447198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:24:39.474140319 +0000 UTC m=+1644.657987654" watchObservedRunningTime="2026-11-02 14:24:39.474447198 +0000 UTC m=+1644.658294535" Nov 02 14:24:39.517121 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:39.517052 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f" podStartSLOduration=2.517034947 podStartE2EDuration="2.517034947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:24:39.514080927 +0000 UTC m=+1644.697928289" watchObservedRunningTime="2026-11-02 14:24:39.517034947 +0000 UTC m=+1644.700882288" Nov 02 14:24:48.429402 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:48.429366 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" Nov 02 14:24:48.507162 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:48.507127 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr"] Nov 02 14:24:50.446566 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:50.446536 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h" Nov 02 14:24:50.446999 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:50.446607 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f" Nov 02 14:24:53.507724 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:53.507673 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" podUID="28800fff-0b1e-40aa-b1b1-9414258d76ed" containerName="payload-processing" containerID="cri-o://4cdac4f04ee337596f286b2d85fa8f4ac0d151cf686702e60bd2127250b14362" gracePeriod=25 Nov 02 14:24:53.807361 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:53.807333 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" Nov 02 14:24:53.893608 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:53.893561 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28800fff-0b1e-40aa-b1b1-9414258d76ed-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28800fff-0b1e-40aa-b1b1-9414258d76ed-routing-overlay-volume\") pod \"28800fff-0b1e-40aa-b1b1-9414258d76ed\" (UID: \"28800fff-0b1e-40aa-b1b1-9414258d76ed\") " Nov 02 14:24:53.893813 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:53.893740 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28800fff-0b1e-40aa-b1b1-9414258d76ed-kube-api-access-pkx47\" (UniqueName: \"kubernetes.io/projected/28800fff-0b1e-40aa-b1b1-9414258d76ed-kube-api-access-pkx47\") pod \"28800fff-0b1e-40aa-b1b1-9414258d76ed\" (UID: \"28800fff-0b1e-40aa-b1b1-9414258d76ed\") " Nov 02 14:24:53.893898 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:53.893870 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28800fff-0b1e-40aa-b1b1-9414258d76ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28800fff-0b1e-40aa-b1b1-9414258d76ed-plugins-config-volume\") pod \"28800fff-0b1e-40aa-b1b1-9414258d76ed\" (UID: \"28800fff-0b1e-40aa-b1b1-9414258d76ed\") " Nov 02 14:24:53.893978 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:53.893929 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28800fff-0b1e-40aa-b1b1-9414258d76ed-provider-credentials\" (UniqueName: \"kubernetes.io/projected/28800fff-0b1e-40aa-b1b1-9414258d76ed-provider-credentials\") pod \"28800fff-0b1e-40aa-b1b1-9414258d76ed\" (UID: \"28800fff-0b1e-40aa-b1b1-9414258d76ed\") " Nov 02 14:24:53.895223 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:53.895187 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28800fff-0b1e-40aa-b1b1-9414258d76ed-routing-overlay-volume" pod "28800fff-0b1e-40aa-b1b1-9414258d76ed" (UID: "28800fff-0b1e-40aa-b1b1-9414258d76ed"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:24:53.896061 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:53.896031 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28800fff-0b1e-40aa-b1b1-9414258d76ed-plugins-config-volume" pod "28800fff-0b1e-40aa-b1b1-9414258d76ed" (UID: "28800fff-0b1e-40aa-b1b1-9414258d76ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:24:53.898386 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:53.898348 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28800fff-0b1e-40aa-b1b1-9414258d76ed-kube-api-access-pkx47" pod "28800fff-0b1e-40aa-b1b1-9414258d76ed" (UID: "28800fff-0b1e-40aa-b1b1-9414258d76ed"). InnerVolumeSpecName "kube-api-access-pkx47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:24:53.898612 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:53.898588 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28800fff-0b1e-40aa-b1b1-9414258d76ed-provider-credentials" pod "28800fff-0b1e-40aa-b1b1-9414258d76ed" (UID: "28800fff-0b1e-40aa-b1b1-9414258d76ed"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:24:53.996254 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:53.996207 4866 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28800fff-0b1e-40aa-b1b1-9414258d76ed-routing-overlay-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:53.996254 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:53.996245 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkx47\" (UniqueName: \"kubernetes.io/projected/28800fff-0b1e-40aa-b1b1-9414258d76ed-kube-api-access-pkx47\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:53.996254 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:53.996256 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28800fff-0b1e-40aa-b1b1-9414258d76ed-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:53.996254 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:53.996265 4866 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/28800fff-0b1e-40aa-b1b1-9414258d76ed-provider-credentials\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:24:54.550524 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:54.550492 4866 generic.go:356] "Generic (PLEG): container finished" podID="28800fff-0b1e-40aa-b1b1-9414258d76ed" containerID="4cdac4f04ee337596f286b2d85fa8f4ac0d151cf686702e60bd2127250b14362" exitCode=0 Nov 02 14:24:54.551029 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:54.550561 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" Nov 02 14:24:54.551029 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:54.550584 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" event={"ID":"28800fff-0b1e-40aa-b1b1-9414258d76ed","Type":"ContainerDied","Data":"4cdac4f04ee337596f286b2d85fa8f4ac0d151cf686702e60bd2127250b14362"} Nov 02 14:24:54.551029 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:54.550654 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr" event={"ID":"28800fff-0b1e-40aa-b1b1-9414258d76ed","Type":"ContainerDied","Data":"e98d15abc53a9a6e368d3eae0f6b0d5f549e52532c28aadfc66223d9fb4bd69e"} Nov 02 14:24:54.551029 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:54.550691 4866 scope.go:122] "RemoveContainer" containerID="4cdac4f04ee337596f286b2d85fa8f4ac0d151cf686702e60bd2127250b14362" Nov 02 14:24:54.561122 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:54.561106 4866 scope.go:122] "RemoveContainer" containerID="4cdac4f04ee337596f286b2d85fa8f4ac0d151cf686702e60bd2127250b14362" Nov 02 14:24:54.561378 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:24:54.561358 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cdac4f04ee337596f286b2d85fa8f4ac0d151cf686702e60bd2127250b14362\": container with ID starting with 4cdac4f04ee337596f286b2d85fa8f4ac0d151cf686702e60bd2127250b14362 not found: ID does not exist" containerID="4cdac4f04ee337596f286b2d85fa8f4ac0d151cf686702e60bd2127250b14362" Nov 02 14:24:54.561439 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:54.561386 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cdac4f04ee337596f286b2d85fa8f4ac0d151cf686702e60bd2127250b14362"} err="failed to get container status \"4cdac4f04ee337596f286b2d85fa8f4ac0d151cf686702e60bd2127250b14362\": rpc error: code = NotFound desc = could not find container \"4cdac4f04ee337596f286b2d85fa8f4ac0d151cf686702e60bd2127250b14362\": container with ID starting with 4cdac4f04ee337596f286b2d85fa8f4ac0d151cf686702e60bd2127250b14362 not found: ID does not exist" Nov 02 14:24:54.603111 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:54.603076 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr"] Nov 02 14:24:54.614572 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:54.614540 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c4dd54948-rb9gr"] Nov 02 14:24:54.947123 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:54.947091 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28800fff-0b1e-40aa-b1b1-9414258d76ed" path="/var/lib/kubelet/pods/28800fff-0b1e-40aa-b1b1-9414258d76ed/volumes" Nov 02 14:24:59.113343 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.113130 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9"] Nov 02 14:24:59.115202 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.115177 4866 cpu_manager.go:414] "containerMap: removing container" podUID="28800fff-0b1e-40aa-b1b1-9414258d76ed" containerName="payload-processing" Nov 02 14:24:59.115202 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.115200 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28800fff-0b1e-40aa-b1b1-9414258d76ed" containerName="payload-processing" Nov 02 14:24:59.115409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.115344 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="28800fff-0b1e-40aa-b1b1-9414258d76ed" containerName="payload-processing" Nov 02 14:24:59.120305 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.120279 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" Nov 02 14:24:59.150353 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.150322 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9"] Nov 02 14:24:59.152157 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.152133 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b3690233-serving-cert\"" Nov 02 14:24:59.274190 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.274143 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c18748-48ff-4a66-8b75-fc7a96a488de-maas-api-tls\") pod \"maas-api-e2e-ait-b3690233-8b559db7b-mnvl9\" (UID: \"71c18748-48ff-4a66-8b75-fc7a96a488de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" Nov 02 14:24:59.274377 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.274224 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c18748-48ff-4a66-8b75-fc7a96a488de-metrics-certs\") pod \"maas-api-e2e-ait-b3690233-8b559db7b-mnvl9\" (UID: \"71c18748-48ff-4a66-8b75-fc7a96a488de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" Nov 02 14:24:59.274452 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.274421 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c18748-48ff-4a66-8b75-fc7a96a488de-openshift-service-ca\") pod \"maas-api-e2e-ait-b3690233-8b559db7b-mnvl9\" (UID: \"71c18748-48ff-4a66-8b75-fc7a96a488de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" Nov 02 14:24:59.274617 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.274596 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km46d\" (UniqueName: \"kubernetes.io/projected/71c18748-48ff-4a66-8b75-fc7a96a488de-kube-api-access-km46d\") pod \"maas-api-e2e-ait-b3690233-8b559db7b-mnvl9\" (UID: \"71c18748-48ff-4a66-8b75-fc7a96a488de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" Nov 02 14:24:59.380892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.380792 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km46d\" (UniqueName: \"kubernetes.io/projected/71c18748-48ff-4a66-8b75-fc7a96a488de-kube-api-access-km46d\") pod \"maas-api-e2e-ait-b3690233-8b559db7b-mnvl9\" (UID: \"71c18748-48ff-4a66-8b75-fc7a96a488de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" Nov 02 14:24:59.381126 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.380910 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c18748-48ff-4a66-8b75-fc7a96a488de-maas-api-tls\") pod \"maas-api-e2e-ait-b3690233-8b559db7b-mnvl9\" (UID: \"71c18748-48ff-4a66-8b75-fc7a96a488de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" Nov 02 14:24:59.381126 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.381040 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c18748-48ff-4a66-8b75-fc7a96a488de-metrics-certs\") pod \"maas-api-e2e-ait-b3690233-8b559db7b-mnvl9\" (UID: \"71c18748-48ff-4a66-8b75-fc7a96a488de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" Nov 02 14:24:59.381255 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.381166 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c18748-48ff-4a66-8b75-fc7a96a488de-openshift-service-ca\") pod \"maas-api-e2e-ait-b3690233-8b559db7b-mnvl9\" (UID: \"71c18748-48ff-4a66-8b75-fc7a96a488de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" Nov 02 14:24:59.382039 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.382010 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c18748-48ff-4a66-8b75-fc7a96a488de-openshift-service-ca\") pod \"maas-api-e2e-ait-b3690233-8b559db7b-mnvl9\" (UID: \"71c18748-48ff-4a66-8b75-fc7a96a488de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" Nov 02 14:24:59.390760 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.390681 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c18748-48ff-4a66-8b75-fc7a96a488de-metrics-certs\") pod \"maas-api-e2e-ait-b3690233-8b559db7b-mnvl9\" (UID: \"71c18748-48ff-4a66-8b75-fc7a96a488de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" Nov 02 14:24:59.408605 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.408572 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c18748-48ff-4a66-8b75-fc7a96a488de-maas-api-tls\") pod \"maas-api-e2e-ait-b3690233-8b559db7b-mnvl9\" (UID: \"71c18748-48ff-4a66-8b75-fc7a96a488de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" Nov 02 14:24:59.417970 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.412648 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km46d\" (UniqueName: \"kubernetes.io/projected/71c18748-48ff-4a66-8b75-fc7a96a488de-kube-api-access-km46d\") pod \"maas-api-e2e-ait-b3690233-8b559db7b-mnvl9\" (UID: \"71c18748-48ff-4a66-8b75-fc7a96a488de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" Nov 02 14:24:59.444563 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.443295 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" Nov 02 14:24:59.782573 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:24:59.782546 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9"] Nov 02 14:24:59.783303 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:24:59.783267 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c18748_48ff_4a66_8b75_fc7a96a488de.slice/crio-7cca17e4b33062f4d2c9aa9dc6fb44e597b25b7478780efad6debc7040d8e07c WatchSource:0}: Error finding container 7cca17e4b33062f4d2c9aa9dc6fb44e597b25b7478780efad6debc7040d8e07c: Status 404 returned error can't find the container with id 7cca17e4b33062f4d2c9aa9dc6fb44e597b25b7478780efad6debc7040d8e07c Nov 02 14:25:00.627239 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:00.627125 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" event={"ID":"71c18748-48ff-4a66-8b75-fc7a96a488de","Type":"ContainerStarted","Data":"3318004e0c35ebca10e300e2d206d193085b6c00416ba707af7987c1199cde4b"} Nov 02 14:25:00.627239 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:00.627192 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" event={"ID":"71c18748-48ff-4a66-8b75-fc7a96a488de","Type":"ContainerStarted","Data":"7cca17e4b33062f4d2c9aa9dc6fb44e597b25b7478780efad6debc7040d8e07c"} Nov 02 14:25:00.627876 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:00.627376 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" Nov 02 14:25:00.672474 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:00.672419 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" podStartSLOduration=1.309719734 podStartE2EDuration="1.672404982s" totalImagesPullingTime="362.685248ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:25:00.670939529 +0000 UTC m=+1665.854786893" watchObservedRunningTime="2026-11-02 14:25:00.672404982 +0000 UTC m=+1665.856252319" Nov 02 14:25:06.028478 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:25:06.028426 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:25:06.666059 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:06.666030 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" Nov 02 14:25:15.699870 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:15.699838 4866 scope.go:122] "RemoveContainer" containerID="bbca98fdaf3cc661c489f339d9922b5e8cf3b613222541e27fe3a003d6987ee3" Nov 02 14:25:32.726168 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:32.726127 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-778c654c9-lrkxv"] Nov 02 14:25:36.009708 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:25:36.009669 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:25:37.727475 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:37.727437 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" podUID="a0582afd-319e-4051-bc1e-b7470824044c" containerName="payload-processing" containerID="cri-o://2c2aa2110badf99b8811c6f41f204943d9f52bf8db5a87296192135c981bca41" gracePeriod=25 Nov 02 14:25:37.930017 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:37.929981 4866 generic.go:356] "Generic (PLEG): container finished" podID="a0582afd-319e-4051-bc1e-b7470824044c" containerID="2c2aa2110badf99b8811c6f41f204943d9f52bf8db5a87296192135c981bca41" exitCode=0 Nov 02 14:25:37.930173 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:37.929982 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" event={"ID":"a0582afd-319e-4051-bc1e-b7470824044c","Type":"ContainerDied","Data":"2c2aa2110badf99b8811c6f41f204943d9f52bf8db5a87296192135c981bca41"} Nov 02 14:25:38.013747 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:38.013718 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" Nov 02 14:25:38.087004 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:38.086975 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0582afd-319e-4051-bc1e-b7470824044c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0582afd-319e-4051-bc1e-b7470824044c-plugins-config-volume\") pod \"a0582afd-319e-4051-bc1e-b7470824044c\" (UID: \"a0582afd-319e-4051-bc1e-b7470824044c\") " Nov 02 14:25:38.087217 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:38.087031 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0582afd-319e-4051-bc1e-b7470824044c-kube-api-access-f759v\" (UniqueName: \"kubernetes.io/projected/a0582afd-319e-4051-bc1e-b7470824044c-kube-api-access-f759v\") pod \"a0582afd-319e-4051-bc1e-b7470824044c\" (UID: \"a0582afd-319e-4051-bc1e-b7470824044c\") " Nov 02 14:25:38.087217 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:38.087131 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0582afd-319e-4051-bc1e-b7470824044c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0582afd-319e-4051-bc1e-b7470824044c-routing-overlay-volume\") pod \"a0582afd-319e-4051-bc1e-b7470824044c\" (UID: \"a0582afd-319e-4051-bc1e-b7470824044c\") " Nov 02 14:25:38.087380 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:38.087225 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0582afd-319e-4051-bc1e-b7470824044c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0582afd-319e-4051-bc1e-b7470824044c-provider-credentials\") pod \"a0582afd-319e-4051-bc1e-b7470824044c\" (UID: \"a0582afd-319e-4051-bc1e-b7470824044c\") " Nov 02 14:25:38.087514 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:38.087484 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0582afd-319e-4051-bc1e-b7470824044c-plugins-config-volume" pod "a0582afd-319e-4051-bc1e-b7470824044c" (UID: "a0582afd-319e-4051-bc1e-b7470824044c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:25:38.087588 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:38.087514 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0582afd-319e-4051-bc1e-b7470824044c-routing-overlay-volume" pod "a0582afd-319e-4051-bc1e-b7470824044c" (UID: "a0582afd-319e-4051-bc1e-b7470824044c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:25:38.088004 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:38.087976 4866 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0582afd-319e-4051-bc1e-b7470824044c-routing-overlay-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:25:38.088113 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:38.088005 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0582afd-319e-4051-bc1e-b7470824044c-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:25:38.089686 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:38.089660 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0582afd-319e-4051-bc1e-b7470824044c-provider-credentials" pod "a0582afd-319e-4051-bc1e-b7470824044c" (UID: "a0582afd-319e-4051-bc1e-b7470824044c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:25:38.090548 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:38.090522 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0582afd-319e-4051-bc1e-b7470824044c-kube-api-access-f759v" pod "a0582afd-319e-4051-bc1e-b7470824044c" (UID: "a0582afd-319e-4051-bc1e-b7470824044c"). InnerVolumeSpecName "kube-api-access-f759v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:25:38.189535 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:38.189495 4866 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0582afd-319e-4051-bc1e-b7470824044c-provider-credentials\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:25:38.189535 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:38.189529 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f759v\" (UniqueName: \"kubernetes.io/projected/a0582afd-319e-4051-bc1e-b7470824044c-kube-api-access-f759v\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:25:38.937625 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:38.937588 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" event={"ID":"a0582afd-319e-4051-bc1e-b7470824044c","Type":"ContainerDied","Data":"ee5b6125a9f92009bb06d957902d07fed140364060654739327ac79fe0960d27"} Nov 02 14:25:38.937625 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:38.937614 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-778c654c9-lrkxv" Nov 02 14:25:38.938223 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:38.937661 4866 scope.go:122] "RemoveContainer" containerID="2c2aa2110badf99b8811c6f41f204943d9f52bf8db5a87296192135c981bca41" Nov 02 14:25:38.991329 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:38.991299 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-778c654c9-lrkxv"] Nov 02 14:25:39.002648 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:39.002620 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-778c654c9-lrkxv"] Nov 02 14:25:40.948568 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:40.948527 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0582afd-319e-4051-bc1e-b7470824044c" path="/var/lib/kubelet/pods/a0582afd-319e-4051-bc1e-b7470824044c/volumes" Nov 02 14:25:42.207416 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:42.207373 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h"] Nov 02 14:25:42.239597 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:42.239561 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f"] Nov 02 14:25:47.208703 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:47.208657 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h" podUID="c83de3cb-ad5f-4e57-83e4-d16913ab66e3" containerName="payload-processing" containerID="cri-o://3fa00b6d0acb50dcd3f8a97be8946e14701e5d2b7254d7b22549f0c592279f6a" gracePeriod=25 Nov 02 14:25:47.240451 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:47.240393 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f" podUID="3ef4069e-ce7d-41e9-9493-2ea93ddbdefa" containerName="payload-pre-processing" containerID="cri-o://2bf4f88f3653d2a68fcb5eaf22de84c7e71d7ea20c163a32c205a93a9c6564b2" gracePeriod=25 Nov 02 14:25:47.518831 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:47.518477 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h" Nov 02 14:25:47.580705 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:47.580663 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f" Nov 02 14:25:47.620440 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:47.620407 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c83de3cb-ad5f-4e57-83e4-d16913ab66e3-kube-api-access-vvjzg\" (UniqueName: \"kubernetes.io/projected/c83de3cb-ad5f-4e57-83e4-d16913ab66e3-kube-api-access-vvjzg\") pod \"c83de3cb-ad5f-4e57-83e4-d16913ab66e3\" (UID: \"c83de3cb-ad5f-4e57-83e4-d16913ab66e3\") " Nov 02 14:25:47.620648 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:47.620612 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c83de3cb-ad5f-4e57-83e4-d16913ab66e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83de3cb-ad5f-4e57-83e4-d16913ab66e3-plugins-config-volume\") pod \"c83de3cb-ad5f-4e57-83e4-d16913ab66e3\" (UID: \"c83de3cb-ad5f-4e57-83e4-d16913ab66e3\") " Nov 02 14:25:47.621078 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:47.621049 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c83de3cb-ad5f-4e57-83e4-d16913ab66e3-plugins-config-volume" pod "c83de3cb-ad5f-4e57-83e4-d16913ab66e3" (UID: "c83de3cb-ad5f-4e57-83e4-d16913ab66e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:25:47.621347 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:47.621329 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83de3cb-ad5f-4e57-83e4-d16913ab66e3-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:25:47.622865 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:47.622839 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83de3cb-ad5f-4e57-83e4-d16913ab66e3-kube-api-access-vvjzg" pod "c83de3cb-ad5f-4e57-83e4-d16913ab66e3" (UID: "c83de3cb-ad5f-4e57-83e4-d16913ab66e3"). InnerVolumeSpecName "kube-api-access-vvjzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:25:47.722688 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:47.722654 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef4069e-ce7d-41e9-9493-2ea93ddbdefa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef4069e-ce7d-41e9-9493-2ea93ddbdefa-plugins-config-volume\") pod \"3ef4069e-ce7d-41e9-9493-2ea93ddbdefa\" (UID: \"3ef4069e-ce7d-41e9-9493-2ea93ddbdefa\") " Nov 02 14:25:47.722883 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:47.722730 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef4069e-ce7d-41e9-9493-2ea93ddbdefa-kube-api-access-n2rmr\" (UniqueName: \"kubernetes.io/projected/3ef4069e-ce7d-41e9-9493-2ea93ddbdefa-kube-api-access-n2rmr\") pod \"3ef4069e-ce7d-41e9-9493-2ea93ddbdefa\" (UID: \"3ef4069e-ce7d-41e9-9493-2ea93ddbdefa\") " Nov 02 14:25:47.723128 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:47.723112 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvjzg\" (UniqueName: \"kubernetes.io/projected/c83de3cb-ad5f-4e57-83e4-d16913ab66e3-kube-api-access-vvjzg\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:25:47.723128 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:47.723115 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef4069e-ce7d-41e9-9493-2ea93ddbdefa-plugins-config-volume" pod "3ef4069e-ce7d-41e9-9493-2ea93ddbdefa" (UID: "3ef4069e-ce7d-41e9-9493-2ea93ddbdefa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:25:47.724852 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:47.724804 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef4069e-ce7d-41e9-9493-2ea93ddbdefa-kube-api-access-n2rmr" pod "3ef4069e-ce7d-41e9-9493-2ea93ddbdefa" (UID: "3ef4069e-ce7d-41e9-9493-2ea93ddbdefa"). InnerVolumeSpecName "kube-api-access-n2rmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:25:47.824581 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:47.824541 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef4069e-ce7d-41e9-9493-2ea93ddbdefa-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:25:47.824581 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:47.824576 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2rmr\" (UniqueName: \"kubernetes.io/projected/3ef4069e-ce7d-41e9-9493-2ea93ddbdefa-kube-api-access-n2rmr\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:25:48.023427 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.023340 4866 generic.go:356] "Generic (PLEG): container finished" podID="c83de3cb-ad5f-4e57-83e4-d16913ab66e3" containerID="3fa00b6d0acb50dcd3f8a97be8946e14701e5d2b7254d7b22549f0c592279f6a" exitCode=0 Nov 02 14:25:48.023427 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.023392 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h" Nov 02 14:25:48.023427 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.023414 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h" event={"ID":"c83de3cb-ad5f-4e57-83e4-d16913ab66e3","Type":"ContainerDied","Data":"3fa00b6d0acb50dcd3f8a97be8946e14701e5d2b7254d7b22549f0c592279f6a"} Nov 02 14:25:48.023685 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.023472 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h" event={"ID":"c83de3cb-ad5f-4e57-83e4-d16913ab66e3","Type":"ContainerDied","Data":"90fbd2b7d7cb46f226d9e56964d3a55256d9be78d089e5d94e6c608f927f8390"} Nov 02 14:25:48.023685 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.023497 4866 scope.go:122] "RemoveContainer" containerID="3fa00b6d0acb50dcd3f8a97be8946e14701e5d2b7254d7b22549f0c592279f6a" Nov 02 14:25:48.025073 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.025050 4866 generic.go:356] "Generic (PLEG): container finished" podID="3ef4069e-ce7d-41e9-9493-2ea93ddbdefa" containerID="2bf4f88f3653d2a68fcb5eaf22de84c7e71d7ea20c163a32c205a93a9c6564b2" exitCode=0 Nov 02 14:25:48.025181 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.025104 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f" Nov 02 14:25:48.025247 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.025105 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f" event={"ID":"3ef4069e-ce7d-41e9-9493-2ea93ddbdefa","Type":"ContainerDied","Data":"2bf4f88f3653d2a68fcb5eaf22de84c7e71d7ea20c163a32c205a93a9c6564b2"} Nov 02 14:25:48.025247 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.025237 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f" event={"ID":"3ef4069e-ce7d-41e9-9493-2ea93ddbdefa","Type":"ContainerDied","Data":"e3a41b03d8de420b4eb9b56c66b80cc5102a24760f8c4f0fa9b77f4d22ae7d60"} Nov 02 14:25:48.034558 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.034534 4866 scope.go:122] "RemoveContainer" containerID="3fa00b6d0acb50dcd3f8a97be8946e14701e5d2b7254d7b22549f0c592279f6a" Nov 02 14:25:48.034926 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:25:48.034903 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa00b6d0acb50dcd3f8a97be8946e14701e5d2b7254d7b22549f0c592279f6a\": container with ID starting with 3fa00b6d0acb50dcd3f8a97be8946e14701e5d2b7254d7b22549f0c592279f6a not found: ID does not exist" containerID="3fa00b6d0acb50dcd3f8a97be8946e14701e5d2b7254d7b22549f0c592279f6a" Nov 02 14:25:48.035078 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.034937 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa00b6d0acb50dcd3f8a97be8946e14701e5d2b7254d7b22549f0c592279f6a"} err="failed to get container status \"3fa00b6d0acb50dcd3f8a97be8946e14701e5d2b7254d7b22549f0c592279f6a\": rpc error: code = NotFound desc = could not find container \"3fa00b6d0acb50dcd3f8a97be8946e14701e5d2b7254d7b22549f0c592279f6a\": container with ID starting with 3fa00b6d0acb50dcd3f8a97be8946e14701e5d2b7254d7b22549f0c592279f6a not found: ID does not exist" Nov 02 14:25:48.035078 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.035013 4866 scope.go:122] "RemoveContainer" containerID="2bf4f88f3653d2a68fcb5eaf22de84c7e71d7ea20c163a32c205a93a9c6564b2" Nov 02 14:25:48.044366 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.044344 4866 scope.go:122] "RemoveContainer" containerID="2bf4f88f3653d2a68fcb5eaf22de84c7e71d7ea20c163a32c205a93a9c6564b2" Nov 02 14:25:48.044694 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:25:48.044674 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf4f88f3653d2a68fcb5eaf22de84c7e71d7ea20c163a32c205a93a9c6564b2\": container with ID starting with 2bf4f88f3653d2a68fcb5eaf22de84c7e71d7ea20c163a32c205a93a9c6564b2 not found: ID does not exist" containerID="2bf4f88f3653d2a68fcb5eaf22de84c7e71d7ea20c163a32c205a93a9c6564b2" Nov 02 14:25:48.044799 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.044703 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf4f88f3653d2a68fcb5eaf22de84c7e71d7ea20c163a32c205a93a9c6564b2"} err="failed to get container status \"2bf4f88f3653d2a68fcb5eaf22de84c7e71d7ea20c163a32c205a93a9c6564b2\": rpc error: code = NotFound desc = could not find container \"2bf4f88f3653d2a68fcb5eaf22de84c7e71d7ea20c163a32c205a93a9c6564b2\": container with ID starting with 2bf4f88f3653d2a68fcb5eaf22de84c7e71d7ea20c163a32c205a93a9c6564b2 not found: ID does not exist" Nov 02 14:25:48.075648 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.075608 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f"] Nov 02 14:25:48.087439 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.087396 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b3690233-6f6c5cd679-7cc8f"] Nov 02 14:25:48.113146 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.113108 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h"] Nov 02 14:25:48.117593 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.117559 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b3690233-d5d6b4bbc-8227h"] Nov 02 14:25:48.979196 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.979154 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef4069e-ce7d-41e9-9493-2ea93ddbdefa" path="/var/lib/kubelet/pods/3ef4069e-ce7d-41e9-9493-2ea93ddbdefa/volumes" Nov 02 14:25:48.979711 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:48.979700 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c83de3cb-ad5f-4e57-83e4-d16913ab66e3" path="/var/lib/kubelet/pods/c83de3cb-ad5f-4e57-83e4-d16913ab66e3/volumes" Nov 02 14:25:52.969179 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:52.969147 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9"] Nov 02 14:25:52.969914 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:52.969882 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" podUID="71c18748-48ff-4a66-8b75-fc7a96a488de" containerName="maas-api" containerID="cri-o://3318004e0c35ebca10e300e2d206d193085b6c00416ba707af7987c1199cde4b" gracePeriod=30 Nov 02 14:25:53.132013 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:53.131974 4866 generic.go:356] "Generic (PLEG): container finished" podID="71c18748-48ff-4a66-8b75-fc7a96a488de" containerID="3318004e0c35ebca10e300e2d206d193085b6c00416ba707af7987c1199cde4b" exitCode=0 Nov 02 14:25:53.132202 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:53.132022 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" event={"ID":"71c18748-48ff-4a66-8b75-fc7a96a488de","Type":"ContainerDied","Data":"3318004e0c35ebca10e300e2d206d193085b6c00416ba707af7987c1199cde4b"} Nov 02 14:25:53.318552 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:53.318525 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" Nov 02 14:25:53.344259 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:53.344016 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c18748-48ff-4a66-8b75-fc7a96a488de-kube-api-access-km46d\" (UniqueName: \"kubernetes.io/projected/71c18748-48ff-4a66-8b75-fc7a96a488de-kube-api-access-km46d\") pod \"71c18748-48ff-4a66-8b75-fc7a96a488de\" (UID: \"71c18748-48ff-4a66-8b75-fc7a96a488de\") " Nov 02 14:25:53.344259 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:53.344098 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71c18748-48ff-4a66-8b75-fc7a96a488de-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c18748-48ff-4a66-8b75-fc7a96a488de-metrics-certs\") pod \"71c18748-48ff-4a66-8b75-fc7a96a488de\" (UID: \"71c18748-48ff-4a66-8b75-fc7a96a488de\") " Nov 02 14:25:53.344259 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:53.344186 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71c18748-48ff-4a66-8b75-fc7a96a488de-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c18748-48ff-4a66-8b75-fc7a96a488de-openshift-service-ca\") pod \"71c18748-48ff-4a66-8b75-fc7a96a488de\" (UID: \"71c18748-48ff-4a66-8b75-fc7a96a488de\") " Nov 02 14:25:53.349972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:53.344479 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71c18748-48ff-4a66-8b75-fc7a96a488de-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c18748-48ff-4a66-8b75-fc7a96a488de-maas-api-tls\") pod \"71c18748-48ff-4a66-8b75-fc7a96a488de\" (UID: \"71c18748-48ff-4a66-8b75-fc7a96a488de\") " Nov 02 14:25:53.349972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:53.349176 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c18748-48ff-4a66-8b75-fc7a96a488de-maas-api-tls" pod "71c18748-48ff-4a66-8b75-fc7a96a488de" (UID: "71c18748-48ff-4a66-8b75-fc7a96a488de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:25:53.349972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:53.349495 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c18748-48ff-4a66-8b75-fc7a96a488de-openshift-service-ca" pod "71c18748-48ff-4a66-8b75-fc7a96a488de" (UID: "71c18748-48ff-4a66-8b75-fc7a96a488de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:25:53.352214 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:53.352178 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c18748-48ff-4a66-8b75-fc7a96a488de-metrics-certs" pod "71c18748-48ff-4a66-8b75-fc7a96a488de" (UID: "71c18748-48ff-4a66-8b75-fc7a96a488de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:25:53.354331 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:53.354306 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c18748-48ff-4a66-8b75-fc7a96a488de-kube-api-access-km46d" pod "71c18748-48ff-4a66-8b75-fc7a96a488de" (UID: "71c18748-48ff-4a66-8b75-fc7a96a488de"). InnerVolumeSpecName "kube-api-access-km46d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:25:53.446769 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:53.446730 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c18748-48ff-4a66-8b75-fc7a96a488de-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:25:53.446769 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:53.446765 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c18748-48ff-4a66-8b75-fc7a96a488de-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:25:53.447006 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:53.446779 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km46d\" (UniqueName: \"kubernetes.io/projected/71c18748-48ff-4a66-8b75-fc7a96a488de-kube-api-access-km46d\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:25:53.447006 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:53.446790 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c18748-48ff-4a66-8b75-fc7a96a488de-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:25:54.139394 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:54.139363 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" Nov 02 14:25:54.139834 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:54.139393 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9" event={"ID":"71c18748-48ff-4a66-8b75-fc7a96a488de","Type":"ContainerDied","Data":"7cca17e4b33062f4d2c9aa9dc6fb44e597b25b7478780efad6debc7040d8e07c"} Nov 02 14:25:54.139834 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:54.139455 4866 scope.go:122] "RemoveContainer" containerID="3318004e0c35ebca10e300e2d206d193085b6c00416ba707af7987c1199cde4b" Nov 02 14:25:54.217708 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:54.217672 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9"] Nov 02 14:25:54.229599 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:54.229572 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b3690233-8b559db7b-mnvl9"] Nov 02 14:25:54.969636 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:54.969409 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71c18748-48ff-4a66-8b75-fc7a96a488de" path="/var/lib/kubelet/pods/71c18748-48ff-4a66-8b75-fc7a96a488de/volumes" Nov 02 14:25:59.359037 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.359000 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8"] Nov 02 14:25:59.359595 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.359582 4866 cpu_manager.go:414] "containerMap: removing container" podUID="c83de3cb-ad5f-4e57-83e4-d16913ab66e3" containerName="payload-processing" Nov 02 14:25:59.359639 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.359596 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c83de3cb-ad5f-4e57-83e4-d16913ab66e3" containerName="payload-processing" Nov 02 14:25:59.359639 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.359611 4866 cpu_manager.go:414] "containerMap: removing container" podUID="3ef4069e-ce7d-41e9-9493-2ea93ddbdefa" containerName="payload-pre-processing" Nov 02 14:25:59.359639 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.359617 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef4069e-ce7d-41e9-9493-2ea93ddbdefa" containerName="payload-pre-processing" Nov 02 14:25:59.359639 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.359629 4866 cpu_manager.go:414] "containerMap: removing container" podUID="71c18748-48ff-4a66-8b75-fc7a96a488de" containerName="maas-api" Nov 02 14:25:59.359639 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.359634 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71c18748-48ff-4a66-8b75-fc7a96a488de" containerName="maas-api" Nov 02 14:25:59.359786 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.359666 4866 cpu_manager.go:414] "containerMap: removing container" podUID="a0582afd-319e-4051-bc1e-b7470824044c" containerName="payload-processing" Nov 02 14:25:59.359786 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.359671 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0582afd-319e-4051-bc1e-b7470824044c" containerName="payload-processing" Nov 02 14:25:59.359786 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.359764 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0582afd-319e-4051-bc1e-b7470824044c" containerName="payload-processing" Nov 02 14:25:59.359786 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.359773 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="71c18748-48ff-4a66-8b75-fc7a96a488de" containerName="maas-api" Nov 02 14:25:59.359786 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.359781 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="c83de3cb-ad5f-4e57-83e4-d16913ab66e3" containerName="payload-processing" Nov 02 14:25:59.359786 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.359786 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef4069e-ce7d-41e9-9493-2ea93ddbdefa" containerName="payload-pre-processing" Nov 02 14:25:59.364509 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.364493 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8" Nov 02 14:25:59.379134 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.379106 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f"] Nov 02 14:25:59.382863 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.382847 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f" Nov 02 14:25:59.392063 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.392041 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8"] Nov 02 14:25:59.399707 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.399673 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-147981-dockercfg-ks27z\"" Nov 02 14:25:59.441251 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.441217 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-147981\"" Nov 02 14:25:59.448235 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.446726 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvxp6\" (UniqueName: \"kubernetes.io/projected/ae1832d0-6f0a-4af9-a644-f07fe725068a-kube-api-access-wvxp6\") pod \"payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8\" (UID: \"ae1832d0-6f0a-4af9-a644-f07fe725068a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8" Nov 02 14:25:59.448235 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.447828 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae1832d0-6f0a-4af9-a644-f07fe725068a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8\" (UID: \"ae1832d0-6f0a-4af9-a644-f07fe725068a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8" Nov 02 14:25:59.497385 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.497344 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f"] Nov 02 14:25:59.570308 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.563103 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvxp6\" (UniqueName: \"kubernetes.io/projected/ae1832d0-6f0a-4af9-a644-f07fe725068a-kube-api-access-wvxp6\") pod \"payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8\" (UID: \"ae1832d0-6f0a-4af9-a644-f07fe725068a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8" Nov 02 14:25:59.570308 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.563258 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6061a0-46f1-4308-b08a-2062ae10dcbf-plugins-config-volume\") pod \"payload-processing-e2e-autores-147981-77f5d75b89-sg26f\" (UID: \"5c6061a0-46f1-4308-b08a-2062ae10dcbf\") " pod="openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f" Nov 02 14:25:59.570308 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.563323 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz89h\" (UniqueName: \"kubernetes.io/projected/5c6061a0-46f1-4308-b08a-2062ae10dcbf-kube-api-access-kz89h\") pod \"payload-processing-e2e-autores-147981-77f5d75b89-sg26f\" (UID: \"5c6061a0-46f1-4308-b08a-2062ae10dcbf\") " pod="openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f" Nov 02 14:25:59.570308 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.563409 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae1832d0-6f0a-4af9-a644-f07fe725068a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8\" (UID: \"ae1832d0-6f0a-4af9-a644-f07fe725068a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8" Nov 02 14:25:59.570308 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.564743 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae1832d0-6f0a-4af9-a644-f07fe725068a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8\" (UID: \"ae1832d0-6f0a-4af9-a644-f07fe725068a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8" Nov 02 14:25:59.600022 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.599856 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvxp6\" (UniqueName: \"kubernetes.io/projected/ae1832d0-6f0a-4af9-a644-f07fe725068a-kube-api-access-wvxp6\") pod \"payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8\" (UID: \"ae1832d0-6f0a-4af9-a644-f07fe725068a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8" Nov 02 14:25:59.664814 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.664734 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6061a0-46f1-4308-b08a-2062ae10dcbf-plugins-config-volume\") pod \"payload-processing-e2e-autores-147981-77f5d75b89-sg26f\" (UID: \"5c6061a0-46f1-4308-b08a-2062ae10dcbf\") " pod="openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f" Nov 02 14:25:59.664814 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.664781 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz89h\" (UniqueName: \"kubernetes.io/projected/5c6061a0-46f1-4308-b08a-2062ae10dcbf-kube-api-access-kz89h\") pod \"payload-processing-e2e-autores-147981-77f5d75b89-sg26f\" (UID: \"5c6061a0-46f1-4308-b08a-2062ae10dcbf\") " pod="openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f" Nov 02 14:25:59.665579 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.665558 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6061a0-46f1-4308-b08a-2062ae10dcbf-plugins-config-volume\") pod \"payload-processing-e2e-autores-147981-77f5d75b89-sg26f\" (UID: \"5c6061a0-46f1-4308-b08a-2062ae10dcbf\") " pod="openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f" Nov 02 14:25:59.679027 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.679000 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8" Nov 02 14:25:59.694008 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.693985 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz89h\" (UniqueName: \"kubernetes.io/projected/5c6061a0-46f1-4308-b08a-2062ae10dcbf-kube-api-access-kz89h\") pod \"payload-processing-e2e-autores-147981-77f5d75b89-sg26f\" (UID: \"5c6061a0-46f1-4308-b08a-2062ae10dcbf\") " pod="openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f" Nov 02 14:25:59.696878 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:25:59.696851 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f" Nov 02 14:26:00.024750 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:00.024715 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8"] Nov 02 14:26:00.025826 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:26:00.025795 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1832d0_6f0a_4af9_a644_f07fe725068a.slice/crio-05088a137524f7f93add485473594a236e2693f9eb1a9a8c5ef3d6bda2a5cd36 WatchSource:0}: Error finding container 05088a137524f7f93add485473594a236e2693f9eb1a9a8c5ef3d6bda2a5cd36: Status 404 returned error can't find the container with id 05088a137524f7f93add485473594a236e2693f9eb1a9a8c5ef3d6bda2a5cd36 Nov 02 14:26:00.040716 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:00.040650 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f"] Nov 02 14:26:00.043200 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:26:00.043166 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6061a0_46f1_4308_b08a_2062ae10dcbf.slice/crio-4edb04483786a069cfc3182a532a93daa6d8d863062bc696fdbb5a6c9a02c44b WatchSource:0}: Error finding container 4edb04483786a069cfc3182a532a93daa6d8d863062bc696fdbb5a6c9a02c44b: Status 404 returned error can't find the container with id 4edb04483786a069cfc3182a532a93daa6d8d863062bc696fdbb5a6c9a02c44b Nov 02 14:26:00.186710 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:00.186671 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f" event={"ID":"5c6061a0-46f1-4308-b08a-2062ae10dcbf","Type":"ContainerStarted","Data":"aad58edd1a5e5edf89b5dd72b5c2feb986ea458be076e6d215e7f22b38b7389f"} Nov 02 14:26:00.186925 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:00.186740 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f" event={"ID":"5c6061a0-46f1-4308-b08a-2062ae10dcbf","Type":"ContainerStarted","Data":"4edb04483786a069cfc3182a532a93daa6d8d863062bc696fdbb5a6c9a02c44b"} Nov 02 14:26:00.186925 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:00.186764 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f" Nov 02 14:26:00.188387 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:00.188360 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8" event={"ID":"ae1832d0-6f0a-4af9-a644-f07fe725068a","Type":"ContainerStarted","Data":"1187d5b978f337c1bb19279aea25dffe2be6a493466d302282d0e388f6641299"} Nov 02 14:26:00.188494 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:00.188404 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8" event={"ID":"ae1832d0-6f0a-4af9-a644-f07fe725068a","Type":"ContainerStarted","Data":"05088a137524f7f93add485473594a236e2693f9eb1a9a8c5ef3d6bda2a5cd36"} Nov 02 14:26:00.188494 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:00.188453 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8" Nov 02 14:26:00.238823 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:00.238768 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f" podStartSLOduration=1.2387548800000001 podStartE2EDuration="1.23875488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:00.237332236 +0000 UTC m=+1725.421179572" watchObservedRunningTime="2026-11-02 14:26:00.23875488 +0000 UTC m=+1725.422602218" Nov 02 14:26:00.279314 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:00.279222 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8" podStartSLOduration=1.279208309 podStartE2EDuration="1.279208309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:00.278725076 +0000 UTC m=+1725.462572414" watchObservedRunningTime="2026-11-02 14:26:00.279208309 +0000 UTC m=+1725.463055645" Nov 02 14:26:01.421816 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.421774 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs"] Nov 02 14:26:01.426061 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.426040 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs" Nov 02 14:26:01.443059 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.443032 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3261bdf2-dockercfg-g2hcb\"" Nov 02 14:26:01.455102 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.455073 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3261bdf2\"" Nov 02 14:26:01.459542 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.459511 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs"] Nov 02 14:26:01.510885 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.510845 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h"] Nov 02 14:26:01.577434 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.577405 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h"] Nov 02 14:26:01.577838 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.577771 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h" Nov 02 14:26:01.603014 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.595851 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4brds\" (UniqueName: \"kubernetes.io/projected/9744e61b-411c-4456-b62a-1aedf5725ef2-kube-api-access-4brds\") pod \"payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs\" (UID: \"9744e61b-411c-4456-b62a-1aedf5725ef2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs" Nov 02 14:26:01.603014 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.595907 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9744e61b-411c-4456-b62a-1aedf5725ef2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs\" (UID: \"9744e61b-411c-4456-b62a-1aedf5725ef2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs" Nov 02 14:26:01.697737 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.697691 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722e3f1a-64e3-4eb1-8171-de4f86ca835c-plugins-config-volume\") pod \"payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h\" (UID: \"722e3f1a-64e3-4eb1-8171-de4f86ca835c\") " pod="openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h" Nov 02 14:26:01.697928 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.697767 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4brds\" (UniqueName: \"kubernetes.io/projected/9744e61b-411c-4456-b62a-1aedf5725ef2-kube-api-access-4brds\") pod \"payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs\" (UID: \"9744e61b-411c-4456-b62a-1aedf5725ef2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs" Nov 02 14:26:01.697928 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.697825 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9744e61b-411c-4456-b62a-1aedf5725ef2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs\" (UID: \"9744e61b-411c-4456-b62a-1aedf5725ef2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs" Nov 02 14:26:01.697928 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.697875 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cb9l\" (UniqueName: \"kubernetes.io/projected/722e3f1a-64e3-4eb1-8171-de4f86ca835c-kube-api-access-4cb9l\") pod \"payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h\" (UID: \"722e3f1a-64e3-4eb1-8171-de4f86ca835c\") " pod="openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h" Nov 02 14:26:01.698545 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.698527 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9744e61b-411c-4456-b62a-1aedf5725ef2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs\" (UID: \"9744e61b-411c-4456-b62a-1aedf5725ef2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs" Nov 02 14:26:01.721443 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.721416 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4brds\" (UniqueName: \"kubernetes.io/projected/9744e61b-411c-4456-b62a-1aedf5725ef2-kube-api-access-4brds\") pod \"payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs\" (UID: \"9744e61b-411c-4456-b62a-1aedf5725ef2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs" Nov 02 14:26:01.740871 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.740844 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs" Nov 02 14:26:01.799392 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.799357 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cb9l\" (UniqueName: \"kubernetes.io/projected/722e3f1a-64e3-4eb1-8171-de4f86ca835c-kube-api-access-4cb9l\") pod \"payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h\" (UID: \"722e3f1a-64e3-4eb1-8171-de4f86ca835c\") " pod="openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h" Nov 02 14:26:01.799935 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.799863 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722e3f1a-64e3-4eb1-8171-de4f86ca835c-plugins-config-volume\") pod \"payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h\" (UID: \"722e3f1a-64e3-4eb1-8171-de4f86ca835c\") " pod="openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h" Nov 02 14:26:01.801124 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.801093 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722e3f1a-64e3-4eb1-8171-de4f86ca835c-plugins-config-volume\") pod \"payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h\" (UID: \"722e3f1a-64e3-4eb1-8171-de4f86ca835c\") " pod="openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h" Nov 02 14:26:01.832058 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.832027 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cb9l\" (UniqueName: \"kubernetes.io/projected/722e3f1a-64e3-4eb1-8171-de4f86ca835c-kube-api-access-4cb9l\") pod \"payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h\" (UID: \"722e3f1a-64e3-4eb1-8171-de4f86ca835c\") " pod="openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h" Nov 02 14:26:01.916783 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:01.916744 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h" Nov 02 14:26:02.046530 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:02.046498 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs"] Nov 02 14:26:02.047386 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:26:02.047356 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9744e61b_411c_4456_b62a_1aedf5725ef2.slice/crio-1949337b7403d6d549be5b11d0258c7b08ffcea65f46eb232980e25a517bb60e WatchSource:0}: Error finding container 1949337b7403d6d549be5b11d0258c7b08ffcea65f46eb232980e25a517bb60e: Status 404 returned error can't find the container with id 1949337b7403d6d549be5b11d0258c7b08ffcea65f46eb232980e25a517bb60e Nov 02 14:26:02.205888 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:02.205849 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs" event={"ID":"9744e61b-411c-4456-b62a-1aedf5725ef2","Type":"ContainerStarted","Data":"0b1b01daf4ef45c4c82ca2973bce5b190805c1d37b83c90b1a822aa35799f4fe"} Nov 02 14:26:02.206071 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:02.205922 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs" event={"ID":"9744e61b-411c-4456-b62a-1aedf5725ef2","Type":"ContainerStarted","Data":"1949337b7403d6d549be5b11d0258c7b08ffcea65f46eb232980e25a517bb60e"} Nov 02 14:26:02.206155 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:02.206115 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs" Nov 02 14:26:02.221051 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:02.220989 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h"] Nov 02 14:26:02.221802 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:26:02.221779 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722e3f1a_64e3_4eb1_8171_de4f86ca835c.slice/crio-022adf951e7dceaa30f41f4f7da7cba3cb3cbaf91ef0186b21f1eba06aa6b3c2 WatchSource:0}: Error finding container 022adf951e7dceaa30f41f4f7da7cba3cb3cbaf91ef0186b21f1eba06aa6b3c2: Status 404 returned error can't find the container with id 022adf951e7dceaa30f41f4f7da7cba3cb3cbaf91ef0186b21f1eba06aa6b3c2 Nov 02 14:26:02.247044 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:02.246992 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs" podStartSLOduration=1.246941785 podStartE2EDuration="1.246941785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:02.245073439 +0000 UTC m=+1727.428920775" watchObservedRunningTime="2026-11-02 14:26:02.246941785 +0000 UTC m=+1727.430789122" Nov 02 14:26:03.215172 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:03.215131 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h" event={"ID":"722e3f1a-64e3-4eb1-8171-de4f86ca835c","Type":"ContainerStarted","Data":"8f6ae79d83081db9d3afda5d17bf4f3fcac89e0feb1e72c98209270810cfab54"} Nov 02 14:26:03.215685 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:03.215193 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h" event={"ID":"722e3f1a-64e3-4eb1-8171-de4f86ca835c","Type":"ContainerStarted","Data":"022adf951e7dceaa30f41f4f7da7cba3cb3cbaf91ef0186b21f1eba06aa6b3c2"} Nov 02 14:26:03.215685 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:03.215306 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h" Nov 02 14:26:03.257533 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:03.257461 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h" podStartSLOduration=2.257296127 podStartE2EDuration="2.257296127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:03.255186701 +0000 UTC m=+1728.439034041" watchObservedRunningTime="2026-11-02 14:26:03.257296127 +0000 UTC m=+1728.441143465" Nov 02 14:26:06.009973 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:26:06.009921 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:26:11.200540 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:11.200510 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8" Nov 02 14:26:11.201053 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:11.200579 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f" Nov 02 14:26:13.218526 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:13.218493 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs" Nov 02 14:26:14.224986 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:14.224941 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h" Nov 02 14:26:20.837566 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:20.837527 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb"] Nov 02 14:26:20.841721 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:20.841697 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" Nov 02 14:26:20.878464 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:20.878434 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-147981-serving-cert\"" Nov 02 14:26:20.883382 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:20.883348 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb"] Nov 02 14:26:20.975093 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:20.975062 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e261eace-605a-418a-b8b6-9c0417d85def-maas-api-tls\") pod \"maas-api-e2e-autores-147981-7b65cb9cc5-f24qb\" (UID: \"e261eace-605a-418a-b8b6-9c0417d85def\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" Nov 02 14:26:20.975272 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:20.975183 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqz4r\" (UniqueName: \"kubernetes.io/projected/e261eace-605a-418a-b8b6-9c0417d85def-kube-api-access-rqz4r\") pod \"maas-api-e2e-autores-147981-7b65cb9cc5-f24qb\" (UID: \"e261eace-605a-418a-b8b6-9c0417d85def\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" Nov 02 14:26:20.975347 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:20.975327 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e261eace-605a-418a-b8b6-9c0417d85def-metrics-certs\") pod \"maas-api-e2e-autores-147981-7b65cb9cc5-f24qb\" (UID: \"e261eace-605a-418a-b8b6-9c0417d85def\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" Nov 02 14:26:20.975385 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:20.975368 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e261eace-605a-418a-b8b6-9c0417d85def-openshift-service-ca\") pod \"maas-api-e2e-autores-147981-7b65cb9cc5-f24qb\" (UID: \"e261eace-605a-418a-b8b6-9c0417d85def\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" Nov 02 14:26:21.076385 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:21.076350 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e261eace-605a-418a-b8b6-9c0417d85def-metrics-certs\") pod \"maas-api-e2e-autores-147981-7b65cb9cc5-f24qb\" (UID: \"e261eace-605a-418a-b8b6-9c0417d85def\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" Nov 02 14:26:21.076385 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:21.076389 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e261eace-605a-418a-b8b6-9c0417d85def-openshift-service-ca\") pod \"maas-api-e2e-autores-147981-7b65cb9cc5-f24qb\" (UID: \"e261eace-605a-418a-b8b6-9c0417d85def\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" Nov 02 14:26:21.076648 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:21.076486 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e261eace-605a-418a-b8b6-9c0417d85def-maas-api-tls\") pod \"maas-api-e2e-autores-147981-7b65cb9cc5-f24qb\" (UID: \"e261eace-605a-418a-b8b6-9c0417d85def\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" Nov 02 14:26:21.076648 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:21.076548 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqz4r\" (UniqueName: \"kubernetes.io/projected/e261eace-605a-418a-b8b6-9c0417d85def-kube-api-access-rqz4r\") pod \"maas-api-e2e-autores-147981-7b65cb9cc5-f24qb\" (UID: \"e261eace-605a-418a-b8b6-9c0417d85def\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" Nov 02 14:26:21.077223 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:21.077195 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e261eace-605a-418a-b8b6-9c0417d85def-openshift-service-ca\") pod \"maas-api-e2e-autores-147981-7b65cb9cc5-f24qb\" (UID: \"e261eace-605a-418a-b8b6-9c0417d85def\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" Nov 02 14:26:21.079004 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:21.078984 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e261eace-605a-418a-b8b6-9c0417d85def-metrics-certs\") pod \"maas-api-e2e-autores-147981-7b65cb9cc5-f24qb\" (UID: \"e261eace-605a-418a-b8b6-9c0417d85def\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" Nov 02 14:26:21.079146 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:21.079123 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e261eace-605a-418a-b8b6-9c0417d85def-maas-api-tls\") pod \"maas-api-e2e-autores-147981-7b65cb9cc5-f24qb\" (UID: \"e261eace-605a-418a-b8b6-9c0417d85def\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" Nov 02 14:26:21.102091 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:21.102025 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqz4r\" (UniqueName: \"kubernetes.io/projected/e261eace-605a-418a-b8b6-9c0417d85def-kube-api-access-rqz4r\") pod \"maas-api-e2e-autores-147981-7b65cb9cc5-f24qb\" (UID: \"e261eace-605a-418a-b8b6-9c0417d85def\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" Nov 02 14:26:21.156072 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:21.155536 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" Nov 02 14:26:21.502067 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:21.502042 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb"] Nov 02 14:26:21.503868 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:26:21.503845 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode261eace_605a_418a_b8b6_9c0417d85def.slice/crio-4e4638c774395da351c6bc2686b9ad371870ff00118419233e67ce4626d9e1ea WatchSource:0}: Error finding container 4e4638c774395da351c6bc2686b9ad371870ff00118419233e67ce4626d9e1ea: Status 404 returned error can't find the container with id 4e4638c774395da351c6bc2686b9ad371870ff00118419233e67ce4626d9e1ea Nov 02 14:26:21.505350 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:21.505332 4866 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:26:22.342365 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:22.342330 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" event={"ID":"e261eace-605a-418a-b8b6-9c0417d85def","Type":"ContainerStarted","Data":"2f6ca8a35bd63fc75d6e30fa42d4e36018cb352aafcf6543538023d0d59f2929"} Nov 02 14:26:22.342800 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:22.342382 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" event={"ID":"e261eace-605a-418a-b8b6-9c0417d85def","Type":"ContainerStarted","Data":"4e4638c774395da351c6bc2686b9ad371870ff00118419233e67ce4626d9e1ea"} Nov 02 14:26:22.342800 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:22.342519 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" Nov 02 14:26:22.388404 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:22.388339 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" podStartSLOduration=1.969291352 podStartE2EDuration="2.388320716s" totalImagesPullingTime="419.029364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:22.3865891 +0000 UTC m=+1747.570436438" watchObservedRunningTime="2026-11-02 14:26:22.388320716 +0000 UTC m=+1747.572168054" Nov 02 14:26:28.354433 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:26:28.354353 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" Nov 02 14:26:36.011029 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:26:36.010992 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:27:06.018676 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:27:06.018632 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:27:10.084020 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:10.083984 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h"] Nov 02 14:27:10.123258 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:10.123223 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs"] Nov 02 14:27:15.085062 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.084997 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h" podUID="722e3f1a-64e3-4eb1-8171-de4f86ca835c" containerName="payload-processing" containerID="cri-o://8f6ae79d83081db9d3afda5d17bf4f3fcac89e0feb1e72c98209270810cfab54" gracePeriod=25 Nov 02 14:27:15.124448 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.124398 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs" podUID="9744e61b-411c-4456-b62a-1aedf5725ef2" containerName="payload-pre-processing" containerID="cri-o://0b1b01daf4ef45c4c82ca2973bce5b190805c1d37b83c90b1a822aa35799f4fe" gracePeriod=25 Nov 02 14:27:15.443051 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.443022 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h" Nov 02 14:27:15.449142 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.449118 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:27:15.469034 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.469012 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:27:15.472420 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.472391 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs" Nov 02 14:27:15.581121 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.581084 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9744e61b-411c-4456-b62a-1aedf5725ef2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9744e61b-411c-4456-b62a-1aedf5725ef2-plugins-config-volume\") pod \"9744e61b-411c-4456-b62a-1aedf5725ef2\" (UID: \"9744e61b-411c-4456-b62a-1aedf5725ef2\") " Nov 02 14:27:15.581121 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.581137 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/722e3f1a-64e3-4eb1-8171-de4f86ca835c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722e3f1a-64e3-4eb1-8171-de4f86ca835c-plugins-config-volume\") pod \"722e3f1a-64e3-4eb1-8171-de4f86ca835c\" (UID: \"722e3f1a-64e3-4eb1-8171-de4f86ca835c\") " Nov 02 14:27:15.581121 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.581164 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722e3f1a-64e3-4eb1-8171-de4f86ca835c-kube-api-access-4cb9l\" (UniqueName: \"kubernetes.io/projected/722e3f1a-64e3-4eb1-8171-de4f86ca835c-kube-api-access-4cb9l\") pod \"722e3f1a-64e3-4eb1-8171-de4f86ca835c\" (UID: \"722e3f1a-64e3-4eb1-8171-de4f86ca835c\") " Nov 02 14:27:15.581489 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.581312 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9744e61b-411c-4456-b62a-1aedf5725ef2-kube-api-access-4brds\" (UniqueName: \"kubernetes.io/projected/9744e61b-411c-4456-b62a-1aedf5725ef2-kube-api-access-4brds\") pod \"9744e61b-411c-4456-b62a-1aedf5725ef2\" (UID: \"9744e61b-411c-4456-b62a-1aedf5725ef2\") " Nov 02 14:27:15.581555 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.581528 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9744e61b-411c-4456-b62a-1aedf5725ef2-plugins-config-volume" pod "9744e61b-411c-4456-b62a-1aedf5725ef2" (UID: "9744e61b-411c-4456-b62a-1aedf5725ef2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:15.581599 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.581573 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/722e3f1a-64e3-4eb1-8171-de4f86ca835c-plugins-config-volume" pod "722e3f1a-64e3-4eb1-8171-de4f86ca835c" (UID: "722e3f1a-64e3-4eb1-8171-de4f86ca835c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:15.581820 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.581804 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9744e61b-411c-4456-b62a-1aedf5725ef2-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:15.581858 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.581824 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722e3f1a-64e3-4eb1-8171-de4f86ca835c-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:15.583457 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.583430 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722e3f1a-64e3-4eb1-8171-de4f86ca835c-kube-api-access-4cb9l" pod "722e3f1a-64e3-4eb1-8171-de4f86ca835c" (UID: "722e3f1a-64e3-4eb1-8171-de4f86ca835c"). InnerVolumeSpecName "kube-api-access-4cb9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:15.583457 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.583450 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9744e61b-411c-4456-b62a-1aedf5725ef2-kube-api-access-4brds" pod "9744e61b-411c-4456-b62a-1aedf5725ef2" (UID: "9744e61b-411c-4456-b62a-1aedf5725ef2"). InnerVolumeSpecName "kube-api-access-4brds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:15.683639 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.683554 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4brds\" (UniqueName: \"kubernetes.io/projected/9744e61b-411c-4456-b62a-1aedf5725ef2-kube-api-access-4brds\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:15.683639 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.683587 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cb9l\" (UniqueName: \"kubernetes.io/projected/722e3f1a-64e3-4eb1-8171-de4f86ca835c-kube-api-access-4cb9l\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:15.762008 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.761973 4866 generic.go:356] "Generic (PLEG): container finished" podID="722e3f1a-64e3-4eb1-8171-de4f86ca835c" containerID="8f6ae79d83081db9d3afda5d17bf4f3fcac89e0feb1e72c98209270810cfab54" exitCode=0 Nov 02 14:27:15.762208 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.762014 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h" Nov 02 14:27:15.762208 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.762038 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h" event={"ID":"722e3f1a-64e3-4eb1-8171-de4f86ca835c","Type":"ContainerDied","Data":"8f6ae79d83081db9d3afda5d17bf4f3fcac89e0feb1e72c98209270810cfab54"} Nov 02 14:27:15.762208 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.762098 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h" event={"ID":"722e3f1a-64e3-4eb1-8171-de4f86ca835c","Type":"ContainerDied","Data":"022adf951e7dceaa30f41f4f7da7cba3cb3cbaf91ef0186b21f1eba06aa6b3c2"} Nov 02 14:27:15.762208 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.762126 4866 scope.go:122] "RemoveContainer" containerID="8f6ae79d83081db9d3afda5d17bf4f3fcac89e0feb1e72c98209270810cfab54" Nov 02 14:27:15.763521 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.763491 4866 generic.go:356] "Generic (PLEG): container finished" podID="9744e61b-411c-4456-b62a-1aedf5725ef2" containerID="0b1b01daf4ef45c4c82ca2973bce5b190805c1d37b83c90b1a822aa35799f4fe" exitCode=0 Nov 02 14:27:15.763626 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.763540 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs" Nov 02 14:27:15.763626 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.763536 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs" event={"ID":"9744e61b-411c-4456-b62a-1aedf5725ef2","Type":"ContainerDied","Data":"0b1b01daf4ef45c4c82ca2973bce5b190805c1d37b83c90b1a822aa35799f4fe"} Nov 02 14:27:15.763739 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.763659 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs" event={"ID":"9744e61b-411c-4456-b62a-1aedf5725ef2","Type":"ContainerDied","Data":"1949337b7403d6d549be5b11d0258c7b08ffcea65f46eb232980e25a517bb60e"} Nov 02 14:27:15.781820 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.781790 4866 scope.go:122] "RemoveContainer" containerID="a643ffc3f4d61a84061e8936fd8e6b9e62108a1facbb565e8b0a68f7580c0359" Nov 02 14:27:15.783857 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.783834 4866 scope.go:122] "RemoveContainer" containerID="8f6ae79d83081db9d3afda5d17bf4f3fcac89e0feb1e72c98209270810cfab54" Nov 02 14:27:15.784177 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:27:15.784156 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f6ae79d83081db9d3afda5d17bf4f3fcac89e0feb1e72c98209270810cfab54\": container with ID starting with 8f6ae79d83081db9d3afda5d17bf4f3fcac89e0feb1e72c98209270810cfab54 not found: ID does not exist" containerID="8f6ae79d83081db9d3afda5d17bf4f3fcac89e0feb1e72c98209270810cfab54" Nov 02 14:27:15.784255 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.784185 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f6ae79d83081db9d3afda5d17bf4f3fcac89e0feb1e72c98209270810cfab54"} err="failed to get container status \"8f6ae79d83081db9d3afda5d17bf4f3fcac89e0feb1e72c98209270810cfab54\": rpc error: code = NotFound desc = could not find container \"8f6ae79d83081db9d3afda5d17bf4f3fcac89e0feb1e72c98209270810cfab54\": container with ID starting with 8f6ae79d83081db9d3afda5d17bf4f3fcac89e0feb1e72c98209270810cfab54 not found: ID does not exist" Nov 02 14:27:15.784255 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.784203 4866 scope.go:122] "RemoveContainer" containerID="0b1b01daf4ef45c4c82ca2973bce5b190805c1d37b83c90b1a822aa35799f4fe" Nov 02 14:27:15.794435 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.794421 4866 scope.go:122] "RemoveContainer" containerID="0b1b01daf4ef45c4c82ca2973bce5b190805c1d37b83c90b1a822aa35799f4fe" Nov 02 14:27:15.794697 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:27:15.794664 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b1b01daf4ef45c4c82ca2973bce5b190805c1d37b83c90b1a822aa35799f4fe\": container with ID starting with 0b1b01daf4ef45c4c82ca2973bce5b190805c1d37b83c90b1a822aa35799f4fe not found: ID does not exist" containerID="0b1b01daf4ef45c4c82ca2973bce5b190805c1d37b83c90b1a822aa35799f4fe" Nov 02 14:27:15.794791 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.794704 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b1b01daf4ef45c4c82ca2973bce5b190805c1d37b83c90b1a822aa35799f4fe"} err="failed to get container status \"0b1b01daf4ef45c4c82ca2973bce5b190805c1d37b83c90b1a822aa35799f4fe\": rpc error: code = NotFound desc = could not find container \"0b1b01daf4ef45c4c82ca2973bce5b190805c1d37b83c90b1a822aa35799f4fe\": container with ID starting with 0b1b01daf4ef45c4c82ca2973bce5b190805c1d37b83c90b1a822aa35799f4fe not found: ID does not exist" Nov 02 14:27:15.830862 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.830830 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs"] Nov 02 14:27:15.837177 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.837153 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3261bdf2-779c6576f9-dm4bs"] Nov 02 14:27:15.875330 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.875294 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h"] Nov 02 14:27:15.880527 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:15.880494 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3261bdf2-657dd64c9f-w8h2h"] Nov 02 14:27:16.947314 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:16.947278 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="722e3f1a-64e3-4eb1-8171-de4f86ca835c" path="/var/lib/kubelet/pods/722e3f1a-64e3-4eb1-8171-de4f86ca835c/volumes" Nov 02 14:27:16.947724 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:16.947706 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9744e61b-411c-4456-b62a-1aedf5725ef2" path="/var/lib/kubelet/pods/9744e61b-411c-4456-b62a-1aedf5725ef2/volumes" Nov 02 14:27:29.822560 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:29.822175 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6"] Nov 02 14:27:29.823754 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:29.823161 4866 cpu_manager.go:414] "containerMap: removing container" podUID="722e3f1a-64e3-4eb1-8171-de4f86ca835c" containerName="payload-processing" Nov 02 14:27:29.823754 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:29.823180 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722e3f1a-64e3-4eb1-8171-de4f86ca835c" containerName="payload-processing" Nov 02 14:27:29.823754 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:29.823202 4866 cpu_manager.go:414] "containerMap: removing container" podUID="9744e61b-411c-4456-b62a-1aedf5725ef2" containerName="payload-pre-processing" Nov 02 14:27:29.823754 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:29.823211 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9744e61b-411c-4456-b62a-1aedf5725ef2" containerName="payload-pre-processing" Nov 02 14:27:29.823754 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:29.823373 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="722e3f1a-64e3-4eb1-8171-de4f86ca835c" containerName="payload-processing" Nov 02 14:27:29.823754 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:29.823387 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="9744e61b-411c-4456-b62a-1aedf5725ef2" containerName="payload-pre-processing" Nov 02 14:27:29.839694 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:29.831429 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6" Nov 02 14:27:29.839694 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:29.837322 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bfcefb-2523-4638-8807-f52e7eb4e86f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6\" (UID: \"29bfcefb-2523-4638-8807-f52e7eb4e86f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6" Nov 02 14:27:29.839694 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:29.837400 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch9qc\" (UniqueName: \"kubernetes.io/projected/29bfcefb-2523-4638-8807-f52e7eb4e86f-kube-api-access-ch9qc\") pod \"payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6\" (UID: \"29bfcefb-2523-4638-8807-f52e7eb4e86f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6" Nov 02 14:27:29.853362 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:29.853314 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-f72bc6\"" Nov 02 14:27:29.853550 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:29.853531 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-f72bc6-dockercfg-zp7xz\"" Nov 02 14:27:29.871144 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:29.870601 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6"] Nov 02 14:27:29.938612 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:29.938572 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bfcefb-2523-4638-8807-f52e7eb4e86f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6\" (UID: \"29bfcefb-2523-4638-8807-f52e7eb4e86f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6" Nov 02 14:27:29.938814 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:29.938655 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch9qc\" (UniqueName: \"kubernetes.io/projected/29bfcefb-2523-4638-8807-f52e7eb4e86f-kube-api-access-ch9qc\") pod \"payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6\" (UID: \"29bfcefb-2523-4638-8807-f52e7eb4e86f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6" Nov 02 14:27:29.939804 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:29.939777 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bfcefb-2523-4638-8807-f52e7eb4e86f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6\" (UID: \"29bfcefb-2523-4638-8807-f52e7eb4e86f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6" Nov 02 14:27:29.962452 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:29.962413 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch9qc\" (UniqueName: \"kubernetes.io/projected/29bfcefb-2523-4638-8807-f52e7eb4e86f-kube-api-access-ch9qc\") pod \"payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6\" (UID: \"29bfcefb-2523-4638-8807-f52e7eb4e86f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6" Nov 02 14:27:30.222853 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:30.222816 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6" Nov 02 14:27:30.526775 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:30.526745 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6"] Nov 02 14:27:30.527909 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:27:30.527884 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bfcefb_2523_4638_8807_f52e7eb4e86f.slice/crio-7e157c70175dcf22188830cc8f4d8e1ded85746e04922e7bb4a3feb55baa5db9 WatchSource:0}: Error finding container 7e157c70175dcf22188830cc8f4d8e1ded85746e04922e7bb4a3feb55baa5db9: Status 404 returned error can't find the container with id 7e157c70175dcf22188830cc8f4d8e1ded85746e04922e7bb4a3feb55baa5db9 Nov 02 14:27:30.927816 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:30.927724 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6" event={"ID":"29bfcefb-2523-4638-8807-f52e7eb4e86f","Type":"ContainerStarted","Data":"82b8dddce3b5191e689286b894ef05d0aac0f369594311a4a97b7bd4a3d6fd85"} Nov 02 14:27:30.927816 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:30.927793 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6" Nov 02 14:27:30.927816 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:30.927815 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6" event={"ID":"29bfcefb-2523-4638-8807-f52e7eb4e86f","Type":"ContainerStarted","Data":"7e157c70175dcf22188830cc8f4d8e1ded85746e04922e7bb4a3feb55baa5db9"} Nov 02 14:27:30.971702 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:30.971644 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6" podStartSLOduration=1.9716259360000001 podStartE2EDuration="1.971625936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:27:30.968511182 +0000 UTC m=+1816.152358513" watchObservedRunningTime="2026-11-02 14:27:30.971625936 +0000 UTC m=+1816.155473321" Nov 02 14:27:36.010530 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:27:36.010492 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:27:38.730492 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:38.730456 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f"] Nov 02 14:27:38.766809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:38.766774 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8"] Nov 02 14:27:38.800251 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:38.800212 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq"] Nov 02 14:27:38.800469 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:38.800442 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" podUID="60bf0c3a-284a-49d1-af2c-34ea34cf44af" containerName="istio-proxy" containerID="cri-o://fe1c8c8795250e0668ff08f44d14227feada0b1896849fa575f41514531233cb" gracePeriod=30 Nov 02 14:27:40.624684 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:40.624636 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2"] Nov 02 14:27:40.625199 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:40.624910 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" podUID="87976167-09be-407f-aa64-6e6a384739ce" containerName="istio-proxy" containerID="cri-o://d981cf9e4f1ed98a35dfa628fef62b3c132aea3eb1b36c020df75bca55c65c19" gracePeriod=30 Nov 02 14:27:41.937862 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:41.937832 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6" Nov 02 14:27:43.458453 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:43.458413 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p"] Nov 02 14:27:43.458960 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:43.458676 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" podUID="88076e68-1465-4d45-9950-48b0b23fdda3" containerName="istio-proxy" containerID="cri-o://30aadfa937cbfc66afce4fbbab4860a76f26a3537ddcf944d45dd25b5c5b5cc3" gracePeriod=30 Nov 02 14:27:43.731803 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:43.731681 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f" podUID="5c6061a0-46f1-4308-b08a-2062ae10dcbf" containerName="payload-processing" containerID="cri-o://aad58edd1a5e5edf89b5dd72b5c2feb986ea458be076e6d215e7f22b38b7389f" gracePeriod=25 Nov 02 14:27:43.767820 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:43.767740 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8" podUID="ae1832d0-6f0a-4af9-a644-f07fe725068a" containerName="payload-pre-processing" containerID="cri-o://1187d5b978f337c1bb19279aea25dffe2be6a493466d302282d0e388f6641299" gracePeriod=25 Nov 02 14:27:44.024998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.023567 4866 generic.go:356] "Generic (PLEG): container finished" podID="5c6061a0-46f1-4308-b08a-2062ae10dcbf" containerID="aad58edd1a5e5edf89b5dd72b5c2feb986ea458be076e6d215e7f22b38b7389f" exitCode=0 Nov 02 14:27:44.024998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.023706 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f" event={"ID":"5c6061a0-46f1-4308-b08a-2062ae10dcbf","Type":"ContainerDied","Data":"aad58edd1a5e5edf89b5dd72b5c2feb986ea458be076e6d215e7f22b38b7389f"} Nov 02 14:27:44.026167 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.026132 4866 generic.go:356] "Generic (PLEG): container finished" podID="ae1832d0-6f0a-4af9-a644-f07fe725068a" containerID="1187d5b978f337c1bb19279aea25dffe2be6a493466d302282d0e388f6641299" exitCode=0 Nov 02 14:27:44.026333 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.026310 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8" event={"ID":"ae1832d0-6f0a-4af9-a644-f07fe725068a","Type":"ContainerDied","Data":"1187d5b978f337c1bb19279aea25dffe2be6a493466d302282d0e388f6641299"} Nov 02 14:27:44.029220 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.028874 4866 generic.go:356] "Generic (PLEG): container finished" podID="60bf0c3a-284a-49d1-af2c-34ea34cf44af" containerID="fe1c8c8795250e0668ff08f44d14227feada0b1896849fa575f41514531233cb" exitCode=0 Nov 02 14:27:44.029220 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.028931 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" event={"ID":"60bf0c3a-284a-49d1-af2c-34ea34cf44af","Type":"ContainerDied","Data":"fe1c8c8795250e0668ff08f44d14227feada0b1896849fa575f41514531233cb"} Nov 02 14:27:44.029220 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.029014 4866 scope.go:122] "RemoveContainer" containerID="4a3880a31c24cd484f8de21f5780d9248715609323f29cbce2fc2d121fb74652" Nov 02 14:27:44.107015 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.104678 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f" Nov 02 14:27:44.142334 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.141499 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8" Nov 02 14:27:44.186621 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.186596 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6061a0-46f1-4308-b08a-2062ae10dcbf-kube-api-access-kz89h\" (UniqueName: \"kubernetes.io/projected/5c6061a0-46f1-4308-b08a-2062ae10dcbf-kube-api-access-kz89h\") pod \"5c6061a0-46f1-4308-b08a-2062ae10dcbf\" (UID: \"5c6061a0-46f1-4308-b08a-2062ae10dcbf\") " Nov 02 14:27:44.186766 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.186679 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c6061a0-46f1-4308-b08a-2062ae10dcbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6061a0-46f1-4308-b08a-2062ae10dcbf-plugins-config-volume\") pod \"5c6061a0-46f1-4308-b08a-2062ae10dcbf\" (UID: \"5c6061a0-46f1-4308-b08a-2062ae10dcbf\") " Nov 02 14:27:44.187313 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.187279 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6061a0-46f1-4308-b08a-2062ae10dcbf-plugins-config-volume" pod "5c6061a0-46f1-4308-b08a-2062ae10dcbf" (UID: "5c6061a0-46f1-4308-b08a-2062ae10dcbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:44.189170 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.189142 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6061a0-46f1-4308-b08a-2062ae10dcbf-kube-api-access-kz89h" pod "5c6061a0-46f1-4308-b08a-2062ae10dcbf" (UID: "5c6061a0-46f1-4308-b08a-2062ae10dcbf"). InnerVolumeSpecName "kube-api-access-kz89h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:44.199461 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.198555 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:27:44.288454 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.288360 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-data\") pod \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " Nov 02 14:27:44.288627 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.288482 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae1832d0-6f0a-4af9-a644-f07fe725068a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae1832d0-6f0a-4af9-a644-f07fe725068a-plugins-config-volume\") pod \"ae1832d0-6f0a-4af9-a644-f07fe725068a\" (UID: \"ae1832d0-6f0a-4af9-a644-f07fe725068a\") " Nov 02 14:27:44.288627 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.288549 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-envoy\") pod \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " Nov 02 14:27:44.288627 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.288583 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istiod-ca-cert\") pod \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " Nov 02 14:27:44.288812 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.288695 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-workload-socket\") pod \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " Nov 02 14:27:44.288812 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.288697 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-data" pod "60bf0c3a-284a-49d1-af2c-34ea34cf44af" (UID: "60bf0c3a-284a-49d1-af2c-34ea34cf44af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:44.288921 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.288809 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-token\" (UniqueName: \"kubernetes.io/projected/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-token\") pod \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " Nov 02 14:27:44.288921 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.288847 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-workload-certs\") pod \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " Nov 02 14:27:44.288921 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.288915 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-credential-socket\") pod \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " Nov 02 14:27:44.289143 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.288959 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-podinfo\") pod \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " Nov 02 14:27:44.289143 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.288984 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60bf0c3a-284a-49d1-af2c-34ea34cf44af-kube-api-access-fmzvh\" (UniqueName: \"kubernetes.io/projected/60bf0c3a-284a-49d1-af2c-34ea34cf44af-kube-api-access-fmzvh\") pod \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\" (UID: \"60bf0c3a-284a-49d1-af2c-34ea34cf44af\") " Nov 02 14:27:44.289143 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.288999 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae1832d0-6f0a-4af9-a644-f07fe725068a-plugins-config-volume" pod "ae1832d0-6f0a-4af9-a644-f07fe725068a" (UID: "ae1832d0-6f0a-4af9-a644-f07fe725068a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:44.289143 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.289051 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1832d0-6f0a-4af9-a644-f07fe725068a-kube-api-access-wvxp6\" (UniqueName: \"kubernetes.io/projected/ae1832d0-6f0a-4af9-a644-f07fe725068a-kube-api-access-wvxp6\") pod \"ae1832d0-6f0a-4af9-a644-f07fe725068a\" (UID: \"ae1832d0-6f0a-4af9-a644-f07fe725068a\") " Nov 02 14:27:44.289336 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.289219 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istiod-ca-cert" pod "60bf0c3a-284a-49d1-af2c-34ea34cf44af" (UID: "60bf0c3a-284a-49d1-af2c-34ea34cf44af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:44.289391 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.289336 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-credential-socket" pod "60bf0c3a-284a-49d1-af2c-34ea34cf44af" (UID: "60bf0c3a-284a-49d1-af2c-34ea34cf44af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:44.289607 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.289518 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-workload-certs" pod "60bf0c3a-284a-49d1-af2c-34ea34cf44af" (UID: "60bf0c3a-284a-49d1-af2c-34ea34cf44af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:44.289607 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.289527 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-workload-socket" pod "60bf0c3a-284a-49d1-af2c-34ea34cf44af" (UID: "60bf0c3a-284a-49d1-af2c-34ea34cf44af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:44.289789 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.289699 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:44.289789 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.289721 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae1832d0-6f0a-4af9-a644-f07fe725068a-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:44.289789 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.289738 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:44.289789 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.289759 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:44.289789 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.289776 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz89h\" (UniqueName: \"kubernetes.io/projected/5c6061a0-46f1-4308-b08a-2062ae10dcbf-kube-api-access-kz89h\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:44.289789 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.289791 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:44.290118 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.289860 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6061a0-46f1-4308-b08a-2062ae10dcbf-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:44.290118 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.289879 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:44.292084 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.292048 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-podinfo" pod "60bf0c3a-284a-49d1-af2c-34ea34cf44af" (UID: "60bf0c3a-284a-49d1-af2c-34ea34cf44af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:27:44.292183 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.292125 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1832d0-6f0a-4af9-a644-f07fe725068a-kube-api-access-wvxp6" pod "ae1832d0-6f0a-4af9-a644-f07fe725068a" (UID: "ae1832d0-6f0a-4af9-a644-f07fe725068a"). InnerVolumeSpecName "kube-api-access-wvxp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:44.292378 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.292357 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-envoy" pod "60bf0c3a-284a-49d1-af2c-34ea34cf44af" (UID: "60bf0c3a-284a-49d1-af2c-34ea34cf44af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:44.292525 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.292398 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60bf0c3a-284a-49d1-af2c-34ea34cf44af-kube-api-access-fmzvh" pod "60bf0c3a-284a-49d1-af2c-34ea34cf44af" (UID: "60bf0c3a-284a-49d1-af2c-34ea34cf44af"). InnerVolumeSpecName "kube-api-access-fmzvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:44.293256 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.293227 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-token" pod "60bf0c3a-284a-49d1-af2c-34ea34cf44af" (UID: "60bf0c3a-284a-49d1-af2c-34ea34cf44af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:44.391369 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.391334 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:44.391369 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.391365 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:44.391369 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.391377 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60bf0c3a-284a-49d1-af2c-34ea34cf44af-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:44.391607 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.391391 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmzvh\" (UniqueName: \"kubernetes.io/projected/60bf0c3a-284a-49d1-af2c-34ea34cf44af-kube-api-access-fmzvh\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:44.391607 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.391404 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvxp6\" (UniqueName: \"kubernetes.io/projected/ae1832d0-6f0a-4af9-a644-f07fe725068a-kube-api-access-wvxp6\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:44.950707 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.950671 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb"] Nov 02 14:27:44.951271 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:44.950879 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" podUID="e261eace-605a-418a-b8b6-9c0417d85def" containerName="maas-api" containerID="cri-o://2f6ca8a35bd63fc75d6e30fa42d4e36018cb352aafcf6543538023d0d59f2929" gracePeriod=30 Nov 02 14:27:45.037457 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.037384 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" Nov 02 14:27:45.037457 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.037421 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq" event={"ID":"60bf0c3a-284a-49d1-af2c-34ea34cf44af","Type":"ContainerDied","Data":"ece69a75386bdc9bc86ea02c6c937814bcd864dbc778b5db531f87ebebb71856"} Nov 02 14:27:45.037685 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.037477 4866 scope.go:122] "RemoveContainer" containerID="fe1c8c8795250e0668ff08f44d14227feada0b1896849fa575f41514531233cb" Nov 02 14:27:45.039537 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.039511 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f" Nov 02 14:27:45.039537 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.039520 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f" event={"ID":"5c6061a0-46f1-4308-b08a-2062ae10dcbf","Type":"ContainerDied","Data":"4edb04483786a069cfc3182a532a93daa6d8d863062bc696fdbb5a6c9a02c44b"} Nov 02 14:27:45.041387 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.041364 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8" event={"ID":"ae1832d0-6f0a-4af9-a644-f07fe725068a","Type":"ContainerDied","Data":"05088a137524f7f93add485473594a236e2693f9eb1a9a8c5ef3d6bda2a5cd36"} Nov 02 14:27:45.041387 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.041380 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8" Nov 02 14:27:45.072581 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.072558 4866 scope.go:122] "RemoveContainer" containerID="aad58edd1a5e5edf89b5dd72b5c2feb986ea458be076e6d215e7f22b38b7389f" Nov 02 14:27:45.083138 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.083111 4866 scope.go:122] "RemoveContainer" containerID="1187d5b978f337c1bb19279aea25dffe2be6a493466d302282d0e388f6641299" Nov 02 14:27:45.094215 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.094174 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq"] Nov 02 14:27:45.101667 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.101630 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-g2wqq"] Nov 02 14:27:45.133735 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.133701 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8"] Nov 02 14:27:45.142277 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.142207 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-147981-7c5d5b86ff-d4mf8"] Nov 02 14:27:45.178884 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.178846 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f"] Nov 02 14:27:45.187120 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.187077 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-147981-77f5d75b89-sg26f"] Nov 02 14:27:45.324660 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.324631 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" Nov 02 14:27:45.438972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.408900 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e261eace-605a-418a-b8b6-9c0417d85def-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e261eace-605a-418a-b8b6-9c0417d85def-maas-api-tls\") pod \"e261eace-605a-418a-b8b6-9c0417d85def\" (UID: \"e261eace-605a-418a-b8b6-9c0417d85def\") " Nov 02 14:27:45.438972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.410023 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e261eace-605a-418a-b8b6-9c0417d85def-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e261eace-605a-418a-b8b6-9c0417d85def-openshift-service-ca\") pod \"e261eace-605a-418a-b8b6-9c0417d85def\" (UID: \"e261eace-605a-418a-b8b6-9c0417d85def\") " Nov 02 14:27:45.438972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.410082 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e261eace-605a-418a-b8b6-9c0417d85def-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e261eace-605a-418a-b8b6-9c0417d85def-metrics-certs\") pod \"e261eace-605a-418a-b8b6-9c0417d85def\" (UID: \"e261eace-605a-418a-b8b6-9c0417d85def\") " Nov 02 14:27:45.438972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.410202 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e261eace-605a-418a-b8b6-9c0417d85def-kube-api-access-rqz4r\" (UniqueName: \"kubernetes.io/projected/e261eace-605a-418a-b8b6-9c0417d85def-kube-api-access-rqz4r\") pod \"e261eace-605a-418a-b8b6-9c0417d85def\" (UID: \"e261eace-605a-418a-b8b6-9c0417d85def\") " Nov 02 14:27:45.438972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.411047 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e261eace-605a-418a-b8b6-9c0417d85def-openshift-service-ca" pod "e261eace-605a-418a-b8b6-9c0417d85def" (UID: "e261eace-605a-418a-b8b6-9c0417d85def"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:45.438972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.412681 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e261eace-605a-418a-b8b6-9c0417d85def-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:45.438972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.418632 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e261eace-605a-418a-b8b6-9c0417d85def-metrics-certs" pod "e261eace-605a-418a-b8b6-9c0417d85def" (UID: "e261eace-605a-418a-b8b6-9c0417d85def"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:27:45.438972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.422356 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e261eace-605a-418a-b8b6-9c0417d85def-kube-api-access-rqz4r" pod "e261eace-605a-418a-b8b6-9c0417d85def" (UID: "e261eace-605a-418a-b8b6-9c0417d85def"). InnerVolumeSpecName "kube-api-access-rqz4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:45.454567 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.454117 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e261eace-605a-418a-b8b6-9c0417d85def-maas-api-tls" pod "e261eace-605a-418a-b8b6-9c0417d85def" (UID: "e261eace-605a-418a-b8b6-9c0417d85def"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:27:45.549841 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.549745 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqz4r\" (UniqueName: \"kubernetes.io/projected/e261eace-605a-418a-b8b6-9c0417d85def-kube-api-access-rqz4r\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:45.549841 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.549779 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e261eace-605a-418a-b8b6-9c0417d85def-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:45.549841 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.549789 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e261eace-605a-418a-b8b6-9c0417d85def-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:45.944020 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:45.943179 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:27:46.049262 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.049224 4866 generic.go:356] "Generic (PLEG): container finished" podID="e261eace-605a-418a-b8b6-9c0417d85def" containerID="2f6ca8a35bd63fc75d6e30fa42d4e36018cb352aafcf6543538023d0d59f2929" exitCode=0 Nov 02 14:27:46.049721 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.049295 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" Nov 02 14:27:46.049721 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.049307 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" event={"ID":"e261eace-605a-418a-b8b6-9c0417d85def","Type":"ContainerDied","Data":"2f6ca8a35bd63fc75d6e30fa42d4e36018cb352aafcf6543538023d0d59f2929"} Nov 02 14:27:46.049721 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.049387 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb" event={"ID":"e261eace-605a-418a-b8b6-9c0417d85def","Type":"ContainerDied","Data":"4e4638c774395da351c6bc2686b9ad371870ff00118419233e67ce4626d9e1ea"} Nov 02 14:27:46.049721 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.049424 4866 scope.go:122] "RemoveContainer" containerID="2f6ca8a35bd63fc75d6e30fa42d4e36018cb352aafcf6543538023d0d59f2929" Nov 02 14:27:46.052419 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.052382 4866 generic.go:356] "Generic (PLEG): container finished" podID="87976167-09be-407f-aa64-6e6a384739ce" containerID="d981cf9e4f1ed98a35dfa628fef62b3c132aea3eb1b36c020df75bca55c65c19" exitCode=0 Nov 02 14:27:46.052537 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.052435 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" Nov 02 14:27:46.052537 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.052494 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" event={"ID":"87976167-09be-407f-aa64-6e6a384739ce","Type":"ContainerDied","Data":"d981cf9e4f1ed98a35dfa628fef62b3c132aea3eb1b36c020df75bca55c65c19"} Nov 02 14:27:46.052634 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.052550 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2" event={"ID":"87976167-09be-407f-aa64-6e6a384739ce","Type":"ContainerDied","Data":"8e3e5fc818d5fe238771250da6b4d8ac15ce6b982371c2f35a5c2d26c55f3fa0"} Nov 02 14:27:46.057085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.057035 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-workload-socket\") pod \"87976167-09be-407f-aa64-6e6a384739ce\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " Nov 02 14:27:46.057175 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.057087 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-istio-data\") pod \"87976167-09be-407f-aa64-6e6a384739ce\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " Nov 02 14:27:46.057175 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.057118 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87976167-09be-407f-aa64-6e6a384739ce-kube-api-access-gb4ld\" (UniqueName: \"kubernetes.io/projected/87976167-09be-407f-aa64-6e6a384739ce-kube-api-access-gb4ld\") pod \"87976167-09be-407f-aa64-6e6a384739ce\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " Nov 02 14:27:46.057175 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.057137 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-istio-envoy\") pod \"87976167-09be-407f-aa64-6e6a384739ce\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " Nov 02 14:27:46.057175 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.057172 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87976167-09be-407f-aa64-6e6a384739ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87976167-09be-407f-aa64-6e6a384739ce-istiod-ca-cert\") pod \"87976167-09be-407f-aa64-6e6a384739ce\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " Nov 02 14:27:46.057382 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.057292 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-credential-socket\") pod \"87976167-09be-407f-aa64-6e6a384739ce\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " Nov 02 14:27:46.057382 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.057329 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87976167-09be-407f-aa64-6e6a384739ce-istio-token\" (UniqueName: \"kubernetes.io/projected/87976167-09be-407f-aa64-6e6a384739ce-istio-token\") pod \"87976167-09be-407f-aa64-6e6a384739ce\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " Nov 02 14:27:46.057487 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.057361 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-workload-socket" pod "87976167-09be-407f-aa64-6e6a384739ce" (UID: "87976167-09be-407f-aa64-6e6a384739ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:46.057487 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.057419 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-istio-data" pod "87976167-09be-407f-aa64-6e6a384739ce" (UID: "87976167-09be-407f-aa64-6e6a384739ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:46.057487 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.057423 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-workload-certs\") pod \"87976167-09be-407f-aa64-6e6a384739ce\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " Nov 02 14:27:46.057487 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.057468 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87976167-09be-407f-aa64-6e6a384739ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87976167-09be-407f-aa64-6e6a384739ce-istio-podinfo\") pod \"87976167-09be-407f-aa64-6e6a384739ce\" (UID: \"87976167-09be-407f-aa64-6e6a384739ce\") " Nov 02 14:27:46.057701 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.057611 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-workload-certs" pod "87976167-09be-407f-aa64-6e6a384739ce" (UID: "87976167-09be-407f-aa64-6e6a384739ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:46.057794 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.057763 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-credential-socket" pod "87976167-09be-407f-aa64-6e6a384739ce" (UID: "87976167-09be-407f-aa64-6e6a384739ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:46.058202 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.058181 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:46.058202 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.058185 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87976167-09be-407f-aa64-6e6a384739ce-istiod-ca-cert" pod "87976167-09be-407f-aa64-6e6a384739ce" (UID: "87976167-09be-407f-aa64-6e6a384739ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:46.058202 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.058204 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:46.058439 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.058218 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:46.058439 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.058231 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:46.060823 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.060790 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87976167-09be-407f-aa64-6e6a384739ce-istio-token" pod "87976167-09be-407f-aa64-6e6a384739ce" (UID: "87976167-09be-407f-aa64-6e6a384739ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:46.060926 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.060820 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87976167-09be-407f-aa64-6e6a384739ce-kube-api-access-gb4ld" pod "87976167-09be-407f-aa64-6e6a384739ce" (UID: "87976167-09be-407f-aa64-6e6a384739ce"). InnerVolumeSpecName "kube-api-access-gb4ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:46.060926 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.060819 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87976167-09be-407f-aa64-6e6a384739ce-istio-podinfo" pod "87976167-09be-407f-aa64-6e6a384739ce" (UID: "87976167-09be-407f-aa64-6e6a384739ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:27:46.061668 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.061642 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-istio-envoy" pod "87976167-09be-407f-aa64-6e6a384739ce" (UID: "87976167-09be-407f-aa64-6e6a384739ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:46.062104 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.062084 4866 scope.go:122] "RemoveContainer" containerID="2f6ca8a35bd63fc75d6e30fa42d4e36018cb352aafcf6543538023d0d59f2929" Nov 02 14:27:46.062405 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:27:46.062384 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f6ca8a35bd63fc75d6e30fa42d4e36018cb352aafcf6543538023d0d59f2929\": container with ID starting with 2f6ca8a35bd63fc75d6e30fa42d4e36018cb352aafcf6543538023d0d59f2929 not found: ID does not exist" containerID="2f6ca8a35bd63fc75d6e30fa42d4e36018cb352aafcf6543538023d0d59f2929" Nov 02 14:27:46.062480 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.062413 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6ca8a35bd63fc75d6e30fa42d4e36018cb352aafcf6543538023d0d59f2929"} err="failed to get container status \"2f6ca8a35bd63fc75d6e30fa42d4e36018cb352aafcf6543538023d0d59f2929\": rpc error: code = NotFound desc = could not find container \"2f6ca8a35bd63fc75d6e30fa42d4e36018cb352aafcf6543538023d0d59f2929\": container with ID starting with 2f6ca8a35bd63fc75d6e30fa42d4e36018cb352aafcf6543538023d0d59f2929 not found: ID does not exist" Nov 02 14:27:46.062480 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.062431 4866 scope.go:122] "RemoveContainer" containerID="d981cf9e4f1ed98a35dfa628fef62b3c132aea3eb1b36c020df75bca55c65c19" Nov 02 14:27:46.088453 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.088418 4866 scope.go:122] "RemoveContainer" containerID="d981cf9e4f1ed98a35dfa628fef62b3c132aea3eb1b36c020df75bca55c65c19" Nov 02 14:27:46.088830 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:27:46.088807 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d981cf9e4f1ed98a35dfa628fef62b3c132aea3eb1b36c020df75bca55c65c19\": container with ID starting with d981cf9e4f1ed98a35dfa628fef62b3c132aea3eb1b36c020df75bca55c65c19 not found: ID does not exist" containerID="d981cf9e4f1ed98a35dfa628fef62b3c132aea3eb1b36c020df75bca55c65c19" Nov 02 14:27:46.088911 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.088840 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d981cf9e4f1ed98a35dfa628fef62b3c132aea3eb1b36c020df75bca55c65c19"} err="failed to get container status \"d981cf9e4f1ed98a35dfa628fef62b3c132aea3eb1b36c020df75bca55c65c19\": rpc error: code = NotFound desc = could not find container \"d981cf9e4f1ed98a35dfa628fef62b3c132aea3eb1b36c020df75bca55c65c19\": container with ID starting with d981cf9e4f1ed98a35dfa628fef62b3c132aea3eb1b36c020df75bca55c65c19 not found: ID does not exist" Nov 02 14:27:46.120974 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.120908 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb"] Nov 02 14:27:46.130774 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.130654 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-147981-7b65cb9cc5-f24qb"] Nov 02 14:27:46.159839 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.159792 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb4ld\" (UniqueName: \"kubernetes.io/projected/87976167-09be-407f-aa64-6e6a384739ce-kube-api-access-gb4ld\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:46.159839 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.159840 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87976167-09be-407f-aa64-6e6a384739ce-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:46.160143 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.159854 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87976167-09be-407f-aa64-6e6a384739ce-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:46.160143 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.159867 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87976167-09be-407f-aa64-6e6a384739ce-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:46.160143 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.159892 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87976167-09be-407f-aa64-6e6a384739ce-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:46.410221 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.410105 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2"] Nov 02 14:27:46.414149 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.414119 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-fvpd2"] Nov 02 14:27:46.946183 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.946154 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c6061a0-46f1-4308-b08a-2062ae10dcbf" path="/var/lib/kubelet/pods/5c6061a0-46f1-4308-b08a-2062ae10dcbf/volumes" Nov 02 14:27:46.946670 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.946656 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60bf0c3a-284a-49d1-af2c-34ea34cf44af" path="/var/lib/kubelet/pods/60bf0c3a-284a-49d1-af2c-34ea34cf44af/volumes" Nov 02 14:27:46.947109 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.947096 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87976167-09be-407f-aa64-6e6a384739ce" path="/var/lib/kubelet/pods/87976167-09be-407f-aa64-6e6a384739ce/volumes" Nov 02 14:27:46.947788 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.947772 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae1832d0-6f0a-4af9-a644-f07fe725068a" path="/var/lib/kubelet/pods/ae1832d0-6f0a-4af9-a644-f07fe725068a/volumes" Nov 02 14:27:46.948221 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:46.948205 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e261eace-605a-418a-b8b6-9c0417d85def" path="/var/lib/kubelet/pods/e261eace-605a-418a-b8b6-9c0417d85def/volumes" Nov 02 14:27:48.723843 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.723788 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:27:48.789166 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.789135 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88076e68-1465-4d45-9950-48b0b23fdda3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88076e68-1465-4d45-9950-48b0b23fdda3-istiod-ca-cert\") pod \"88076e68-1465-4d45-9950-48b0b23fdda3\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " Nov 02 14:27:48.789166 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.789168 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88076e68-1465-4d45-9950-48b0b23fdda3-istio-token\" (UniqueName: \"kubernetes.io/projected/88076e68-1465-4d45-9950-48b0b23fdda3-istio-token\") pod \"88076e68-1465-4d45-9950-48b0b23fdda3\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " Nov 02 14:27:48.789354 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.789193 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-istio-data\") pod \"88076e68-1465-4d45-9950-48b0b23fdda3\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " Nov 02 14:27:48.789354 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.789230 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-workload-socket\") pod \"88076e68-1465-4d45-9950-48b0b23fdda3\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " Nov 02 14:27:48.789354 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.789295 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-credential-socket\") pod \"88076e68-1465-4d45-9950-48b0b23fdda3\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " Nov 02 14:27:48.789354 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.789348 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88076e68-1465-4d45-9950-48b0b23fdda3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88076e68-1465-4d45-9950-48b0b23fdda3-istio-podinfo\") pod \"88076e68-1465-4d45-9950-48b0b23fdda3\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " Nov 02 14:27:48.789526 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.789374 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-istio-envoy\") pod \"88076e68-1465-4d45-9950-48b0b23fdda3\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " Nov 02 14:27:48.789526 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.789478 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-workload-certs\") pod \"88076e68-1465-4d45-9950-48b0b23fdda3\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " Nov 02 14:27:48.789526 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.789488 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-istio-data" pod "88076e68-1465-4d45-9950-48b0b23fdda3" (UID: "88076e68-1465-4d45-9950-48b0b23fdda3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:48.789526 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.789522 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88076e68-1465-4d45-9950-48b0b23fdda3-kube-api-access-qghkm\" (UniqueName: \"kubernetes.io/projected/88076e68-1465-4d45-9950-48b0b23fdda3-kube-api-access-qghkm\") pod \"88076e68-1465-4d45-9950-48b0b23fdda3\" (UID: \"88076e68-1465-4d45-9950-48b0b23fdda3\") " Nov 02 14:27:48.789732 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.789565 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-workload-socket" pod "88076e68-1465-4d45-9950-48b0b23fdda3" (UID: "88076e68-1465-4d45-9950-48b0b23fdda3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:48.789732 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.789626 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88076e68-1465-4d45-9950-48b0b23fdda3-istiod-ca-cert" pod "88076e68-1465-4d45-9950-48b0b23fdda3" (UID: "88076e68-1465-4d45-9950-48b0b23fdda3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:48.789841 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.789732 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-workload-certs" pod "88076e68-1465-4d45-9950-48b0b23fdda3" (UID: "88076e68-1465-4d45-9950-48b0b23fdda3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:48.789925 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.789904 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-credential-socket" pod "88076e68-1465-4d45-9950-48b0b23fdda3" (UID: "88076e68-1465-4d45-9950-48b0b23fdda3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:48.790111 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.790090 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:48.790187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.790116 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:48.790187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.790134 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88076e68-1465-4d45-9950-48b0b23fdda3-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:48.790187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.790147 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:48.790187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.790160 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:48.791699 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.791659 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88076e68-1465-4d45-9950-48b0b23fdda3-istio-podinfo" pod "88076e68-1465-4d45-9950-48b0b23fdda3" (UID: "88076e68-1465-4d45-9950-48b0b23fdda3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:27:48.791812 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.791699 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88076e68-1465-4d45-9950-48b0b23fdda3-istio-token" pod "88076e68-1465-4d45-9950-48b0b23fdda3" (UID: "88076e68-1465-4d45-9950-48b0b23fdda3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:48.791812 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.791759 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-istio-envoy" pod "88076e68-1465-4d45-9950-48b0b23fdda3" (UID: "88076e68-1465-4d45-9950-48b0b23fdda3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:48.791921 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.791884 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88076e68-1465-4d45-9950-48b0b23fdda3-kube-api-access-qghkm" pod "88076e68-1465-4d45-9950-48b0b23fdda3" (UID: "88076e68-1465-4d45-9950-48b0b23fdda3"). InnerVolumeSpecName "kube-api-access-qghkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:48.891660 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.891631 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88076e68-1465-4d45-9950-48b0b23fdda3-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:48.891660 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.891658 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88076e68-1465-4d45-9950-48b0b23fdda3-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:48.891660 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.891669 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88076e68-1465-4d45-9950-48b0b23fdda3-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:48.891908 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:48.891678 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qghkm\" (UniqueName: \"kubernetes.io/projected/88076e68-1465-4d45-9950-48b0b23fdda3-kube-api-access-qghkm\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:27:49.076042 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:49.075926 4866 generic.go:356] "Generic (PLEG): container finished" podID="88076e68-1465-4d45-9950-48b0b23fdda3" containerID="30aadfa937cbfc66afce4fbbab4860a76f26a3537ddcf944d45dd25b5c5b5cc3" exitCode=0 Nov 02 14:27:49.076042 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:49.075987 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" Nov 02 14:27:49.076042 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:49.076010 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" event={"ID":"88076e68-1465-4d45-9950-48b0b23fdda3","Type":"ContainerDied","Data":"30aadfa937cbfc66afce4fbbab4860a76f26a3537ddcf944d45dd25b5c5b5cc3"} Nov 02 14:27:49.076320 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:49.076078 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p" event={"ID":"88076e68-1465-4d45-9950-48b0b23fdda3","Type":"ContainerDied","Data":"a02ae1e3a83232d9528e67e4b75dfd5994edecb197798f51ad7a314385c2392c"} Nov 02 14:27:49.076320 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:49.076108 4866 scope.go:122] "RemoveContainer" containerID="30aadfa937cbfc66afce4fbbab4860a76f26a3537ddcf944d45dd25b5c5b5cc3" Nov 02 14:27:49.086139 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:49.086109 4866 scope.go:122] "RemoveContainer" containerID="30aadfa937cbfc66afce4fbbab4860a76f26a3537ddcf944d45dd25b5c5b5cc3" Nov 02 14:27:49.086488 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:27:49.086465 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30aadfa937cbfc66afce4fbbab4860a76f26a3537ddcf944d45dd25b5c5b5cc3\": container with ID starting with 30aadfa937cbfc66afce4fbbab4860a76f26a3537ddcf944d45dd25b5c5b5cc3 not found: ID does not exist" containerID="30aadfa937cbfc66afce4fbbab4860a76f26a3537ddcf944d45dd25b5c5b5cc3" Nov 02 14:27:49.086566 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:49.086501 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30aadfa937cbfc66afce4fbbab4860a76f26a3537ddcf944d45dd25b5c5b5cc3"} err="failed to get container status \"30aadfa937cbfc66afce4fbbab4860a76f26a3537ddcf944d45dd25b5c5b5cc3\": rpc error: code = NotFound desc = could not find container \"30aadfa937cbfc66afce4fbbab4860a76f26a3537ddcf944d45dd25b5c5b5cc3\": container with ID starting with 30aadfa937cbfc66afce4fbbab4860a76f26a3537ddcf944d45dd25b5c5b5cc3 not found: ID does not exist" Nov 02 14:27:49.132348 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:49.132314 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p"] Nov 02 14:27:49.139284 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:49.139258 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-7q22p"] Nov 02 14:27:50.380016 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.379976 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4"] Nov 02 14:27:50.380877 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.380850 4866 cpu_manager.go:414] "containerMap: removing container" podUID="e261eace-605a-418a-b8b6-9c0417d85def" containerName="maas-api" Nov 02 14:27:50.380965 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.380880 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e261eace-605a-418a-b8b6-9c0417d85def" containerName="maas-api" Nov 02 14:27:50.380965 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.380914 4866 cpu_manager.go:414] "containerMap: removing container" podUID="ae1832d0-6f0a-4af9-a644-f07fe725068a" containerName="payload-pre-processing" Nov 02 14:27:50.380965 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.380922 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1832d0-6f0a-4af9-a644-f07fe725068a" containerName="payload-pre-processing" Nov 02 14:27:50.380965 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.380957 4866 cpu_manager.go:414] "containerMap: removing container" podUID="60bf0c3a-284a-49d1-af2c-34ea34cf44af" containerName="istio-proxy" Nov 02 14:27:50.380965 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.380965 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60bf0c3a-284a-49d1-af2c-34ea34cf44af" containerName="istio-proxy" Nov 02 14:27:50.381189 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.380991 4866 cpu_manager.go:414] "containerMap: removing container" podUID="5c6061a0-46f1-4308-b08a-2062ae10dcbf" containerName="payload-processing" Nov 02 14:27:50.381189 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.380998 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6061a0-46f1-4308-b08a-2062ae10dcbf" containerName="payload-processing" Nov 02 14:27:50.381189 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.381007 4866 cpu_manager.go:414] "containerMap: removing container" podUID="60bf0c3a-284a-49d1-af2c-34ea34cf44af" containerName="istio-proxy" Nov 02 14:27:50.381189 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.381016 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60bf0c3a-284a-49d1-af2c-34ea34cf44af" containerName="istio-proxy" Nov 02 14:27:50.381189 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.381038 4866 cpu_manager.go:414] "containerMap: removing container" podUID="88076e68-1465-4d45-9950-48b0b23fdda3" containerName="istio-proxy" Nov 02 14:27:50.381189 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.381045 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88076e68-1465-4d45-9950-48b0b23fdda3" containerName="istio-proxy" Nov 02 14:27:50.381189 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.381065 4866 cpu_manager.go:414] "containerMap: removing container" podUID="87976167-09be-407f-aa64-6e6a384739ce" containerName="istio-proxy" Nov 02 14:27:50.381189 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.381071 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87976167-09be-407f-aa64-6e6a384739ce" containerName="istio-proxy" Nov 02 14:27:50.381409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.381213 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="88076e68-1465-4d45-9950-48b0b23fdda3" containerName="istio-proxy" Nov 02 14:27:50.381409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.381227 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="e261eace-605a-418a-b8b6-9c0417d85def" containerName="maas-api" Nov 02 14:27:50.381409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.381237 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6061a0-46f1-4308-b08a-2062ae10dcbf" containerName="payload-processing" Nov 02 14:27:50.381409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.381247 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="60bf0c3a-284a-49d1-af2c-34ea34cf44af" containerName="istio-proxy" Nov 02 14:27:50.381409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.381262 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="87976167-09be-407f-aa64-6e6a384739ce" containerName="istio-proxy" Nov 02 14:27:50.381409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.381274 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1832d0-6f0a-4af9-a644-f07fe725068a" containerName="payload-pre-processing" Nov 02 14:27:50.386969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.386923 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.406913 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.406889 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-5a4525-openshift-default-dockercfg-b5mjk\"" Nov 02 14:27:50.411532 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.411513 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fph7d\" (UniqueName: \"kubernetes.io/projected/c91b5d92-2a59-46c7-ac62-f398a392a8ff-kube-api-access-fph7d\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.411585 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.411559 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-workload-socket\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.411660 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.411637 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istiod-ca-cert\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.411721 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.411706 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-envoy\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.411760 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.411746 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-credential-socket\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.411794 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.411770 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-data\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.411844 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.411831 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-workload-certs\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.411879 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.411852 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-token\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.411917 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.411876 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-podinfo\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.422922 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.422894 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4"] Nov 02 14:27:50.514972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.514562 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istiod-ca-cert\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.514972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.514618 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-envoy\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.514972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.514668 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-credential-socket\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.514972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.514691 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-data\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.514972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.514718 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-workload-certs\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.514972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.514743 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-token\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.514972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.514779 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-podinfo\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.515482 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.514994 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fph7d\" (UniqueName: \"kubernetes.io/projected/c91b5d92-2a59-46c7-ac62-f398a392a8ff-kube-api-access-fph7d\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.515482 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.515067 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-workload-socket\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.518608 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.515606 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-workload-socket\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.518608 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.516618 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istiod-ca-cert\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.518608 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.517276 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-credential-socket\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.518608 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.517548 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-data\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.522045 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.519875 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-workload-certs\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.542008 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.539669 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-podinfo\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.587968 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.567655 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fph7d\" (UniqueName: \"kubernetes.io/projected/c91b5d92-2a59-46c7-ac62-f398a392a8ff-kube-api-access-fph7d\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.587968 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.567729 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-token\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.587968 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.574286 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-envoy\") pod \"e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.702585 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.702547 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:50.947375 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:50.947342 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88076e68-1465-4d45-9950-48b0b23fdda3" path="/var/lib/kubelet/pods/88076e68-1465-4d45-9950-48b0b23fdda3/volumes" Nov 02 14:27:51.021044 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:51.021020 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4"] Nov 02 14:27:51.029892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:51.029849 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:27:51.030084 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:51.029963 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:27:51.030084 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:51.029996 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:27:51.092042 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:51.091996 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" event={"ID":"c91b5d92-2a59-46c7-ac62-f398a392a8ff","Type":"ContainerStarted","Data":"b8e3c9ac53b2cbcb27dbcc81048b83ab56705df8c5687c3e69b835196b59e57e"} Nov 02 14:27:52.099103 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:52.099064 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" event={"ID":"c91b5d92-2a59-46c7-ac62-f398a392a8ff","Type":"ContainerStarted","Data":"d63f03129bbf648a49d5216dbedb2e3213b6348960d81522142b0b3b31842d0b"} Nov 02 14:27:52.170602 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:52.170548 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" podStartSLOduration=2.170532898 podStartE2EDuration="2.170532898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:27:52.156722283 +0000 UTC m=+1837.340569619" watchObservedRunningTime="2026-11-02 14:27:52.170532898 +0000 UTC m=+1837.354380234" Nov 02 14:27:52.702749 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:52.702712 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:52.708808 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:52.708785 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:53.105316 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:53.105224 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:53.106200 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:53.106182 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:27:53.512226 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:53.512192 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt"] Nov 02 14:27:53.513151 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:53.513134 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="60bf0c3a-284a-49d1-af2c-34ea34cf44af" containerName="istio-proxy" Nov 02 14:27:53.517037 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:53.517019 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" Nov 02 14:27:53.535293 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:53.535240 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-f72bc6-serving-cert\"" Nov 02 14:27:53.542629 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:53.542590 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt"] Nov 02 14:27:53.557441 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:53.557404 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt\" (UID: \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" Nov 02 14:27:53.557636 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:53.557478 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5pxx\" (UniqueName: \"kubernetes.io/projected/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-kube-api-access-m5pxx\") pod \"maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt\" (UID: \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" Nov 02 14:27:53.557636 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:53.557607 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-metrics-certs\") pod \"maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt\" (UID: \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" Nov 02 14:27:53.557761 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:53.557667 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt\" (UID: \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" Nov 02 14:27:53.659074 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:53.659034 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-metrics-certs\") pod \"maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt\" (UID: \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" Nov 02 14:27:53.659257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:53.659096 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt\" (UID: \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" Nov 02 14:27:53.659257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:53.659176 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt\" (UID: \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" Nov 02 14:27:53.659257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:53.659207 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5pxx\" (UniqueName: \"kubernetes.io/projected/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-kube-api-access-m5pxx\") pod \"maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt\" (UID: \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" Nov 02 14:27:53.659359 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:27:53.659331 4866 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-serving-cert: secret "maas-api-e2e-worker-w1-f72bc6-serving-cert" not found Nov 02 14:27:53.659438 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:27:53.659424 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-maas-api-tls podName:b2a9fbab-fdf4-4d57-9aae-74842fb231c9 nodeName:}" failed. No retries permitted until 2026-11-02 14:27:54.159402071 +0000 UTC m=+1839.343249403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-maas-api-tls") pod "maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" (UID: "b2a9fbab-fdf4-4d57-9aae-74842fb231c9") : secret "maas-api-e2e-worker-w1-f72bc6-serving-cert" not found Nov 02 14:27:53.659872 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:53.659854 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt\" (UID: \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" Nov 02 14:27:53.661576 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:53.661557 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-metrics-certs\") pod \"maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt\" (UID: \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" Nov 02 14:27:53.684268 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:53.684023 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5pxx\" (UniqueName: \"kubernetes.io/projected/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-kube-api-access-m5pxx\") pod \"maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt\" (UID: \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" Nov 02 14:27:54.166645 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:54.166604 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt\" (UID: \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" Nov 02 14:27:54.174836 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:54.170224 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt\" (UID: \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" Nov 02 14:27:54.436593 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:54.436466 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" Nov 02 14:27:54.759417 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:54.758515 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt"] Nov 02 14:27:54.761596 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:27:54.761553 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a9fbab_fdf4_4d57_9aae_74842fb231c9.slice/crio-e54b93f7be1a517a1037b75d1531c7117667614c22bdc7a9e16e03b006f1fc67 WatchSource:0}: Error finding container e54b93f7be1a517a1037b75d1531c7117667614c22bdc7a9e16e03b006f1fc67: Status 404 returned error can't find the container with id e54b93f7be1a517a1037b75d1531c7117667614c22bdc7a9e16e03b006f1fc67 Nov 02 14:27:55.118211 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:55.118174 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" event={"ID":"b2a9fbab-fdf4-4d57-9aae-74842fb231c9","Type":"ContainerStarted","Data":"e54b93f7be1a517a1037b75d1531c7117667614c22bdc7a9e16e03b006f1fc67"} Nov 02 14:27:55.763213 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:55.763180 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s"] Nov 02 14:27:55.763657 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:55.763369 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" podUID="9a1faca5-3a0e-4a93-afda-78b9aa799050" containerName="istio-proxy" containerID="cri-o://84ef9de0c7303f2610b22e64dc5d5c169d4839772c09628d0e4a45fabf089fdd" gracePeriod=30 Nov 02 14:27:56.127018 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:56.126894 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" event={"ID":"b2a9fbab-fdf4-4d57-9aae-74842fb231c9","Type":"ContainerStarted","Data":"3436d2d525ce5d2a76ab378ae2f724666587865e96b2ff284292cc4bdbb6b1a1"} Nov 02 14:27:56.127018 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:56.126969 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" Nov 02 14:27:56.172426 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:56.172367 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" podStartSLOduration=2.84235468 podStartE2EDuration="3.172349235s" totalImagesPullingTime="329.994555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:27:56.170301601 +0000 UTC m=+1841.354148938" watchObservedRunningTime="2026-11-02 14:27:56.172349235 +0000 UTC m=+1841.356196587" Nov 02 14:27:56.974179 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:56.974142 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl"] Nov 02 14:27:56.978433 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:56.978409 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl" Nov 02 14:27:56.995882 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:56.995850 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-5a4525-dockercfg-6928q\"" Nov 02 14:27:56.998698 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:56.998668 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl"] Nov 02 14:27:57.037134 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.037088 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-5a4525\"" Nov 02 14:27:57.189559 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.189510 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427c3914-9b8b-467f-bcb6-ccb99ee098cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl\" (UID: \"427c3914-9b8b-467f-bcb6-ccb99ee098cf\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl" Nov 02 14:27:57.193627 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.193203 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjr5s\" (UniqueName: \"kubernetes.io/projected/427c3914-9b8b-467f-bcb6-ccb99ee098cf-kube-api-access-cjr5s\") pod \"payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl\" (UID: \"427c3914-9b8b-467f-bcb6-ccb99ee098cf\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl" Nov 02 14:27:57.196247 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.196208 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf"] Nov 02 14:27:57.236049 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.235717 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf"] Nov 02 14:27:57.236049 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.235806 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf" Nov 02 14:27:57.295646 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.295605 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjr5s\" (UniqueName: \"kubernetes.io/projected/427c3914-9b8b-467f-bcb6-ccb99ee098cf-kube-api-access-cjr5s\") pod \"payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl\" (UID: \"427c3914-9b8b-467f-bcb6-ccb99ee098cf\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl" Nov 02 14:27:57.295874 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.295850 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e5e98d-359e-4417-a474-c3678ab12022-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf\" (UID: \"c2e5e98d-359e-4417-a474-c3678ab12022\") " pod="openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf" Nov 02 14:27:57.295994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.295968 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxpls\" (UniqueName: \"kubernetes.io/projected/c2e5e98d-359e-4417-a474-c3678ab12022-kube-api-access-vxpls\") pod \"payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf\" (UID: \"c2e5e98d-359e-4417-a474-c3678ab12022\") " pod="openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf" Nov 02 14:27:57.296151 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.296132 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427c3914-9b8b-467f-bcb6-ccb99ee098cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl\" (UID: \"427c3914-9b8b-467f-bcb6-ccb99ee098cf\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl" Nov 02 14:27:57.296821 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.296803 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427c3914-9b8b-467f-bcb6-ccb99ee098cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl\" (UID: \"427c3914-9b8b-467f-bcb6-ccb99ee098cf\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl" Nov 02 14:27:57.319560 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.319524 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjr5s\" (UniqueName: \"kubernetes.io/projected/427c3914-9b8b-467f-bcb6-ccb99ee098cf-kube-api-access-cjr5s\") pod \"payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl\" (UID: \"427c3914-9b8b-467f-bcb6-ccb99ee098cf\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl" Nov 02 14:27:57.424835 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.424740 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e5e98d-359e-4417-a474-c3678ab12022-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf\" (UID: \"c2e5e98d-359e-4417-a474-c3678ab12022\") " pod="openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf" Nov 02 14:27:57.425176 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.425152 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxpls\" (UniqueName: \"kubernetes.io/projected/c2e5e98d-359e-4417-a474-c3678ab12022-kube-api-access-vxpls\") pod \"payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf\" (UID: \"c2e5e98d-359e-4417-a474-c3678ab12022\") " pod="openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf" Nov 02 14:27:57.426652 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.426626 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e5e98d-359e-4417-a474-c3678ab12022-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf\" (UID: \"c2e5e98d-359e-4417-a474-c3678ab12022\") " pod="openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf" Nov 02 14:27:57.474965 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.474659 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxpls\" (UniqueName: \"kubernetes.io/projected/c2e5e98d-359e-4417-a474-c3678ab12022-kube-api-access-vxpls\") pod \"payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf\" (UID: \"c2e5e98d-359e-4417-a474-c3678ab12022\") " pod="openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf" Nov 02 14:27:57.550467 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.550374 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf" Nov 02 14:27:57.591815 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.591785 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl" Nov 02 14:27:57.902366 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.898518 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf"] Nov 02 14:27:57.980289 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:57.978686 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl"] Nov 02 14:27:58.242412 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:58.242369 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf" event={"ID":"c2e5e98d-359e-4417-a474-c3678ab12022","Type":"ContainerStarted","Data":"8ccd098fcf67d43285995c2e9410a210e8509636f32b2af47b3dcb030792469d"} Nov 02 14:27:58.242590 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:58.242445 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf" event={"ID":"c2e5e98d-359e-4417-a474-c3678ab12022","Type":"ContainerStarted","Data":"c907a5638053dd9f6905b5d3f840e88a0af8f0396087cfca7c47556d20f7ab4d"} Nov 02 14:27:58.242590 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:58.242565 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf" Nov 02 14:27:58.244455 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:58.244425 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl" event={"ID":"427c3914-9b8b-467f-bcb6-ccb99ee098cf","Type":"ContainerStarted","Data":"87d9c913b7e1fa0faa47f0a777953f201c0c4e429487fa15003149426ea37937"} Nov 02 14:27:58.244606 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:58.244473 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl" event={"ID":"427c3914-9b8b-467f-bcb6-ccb99ee098cf","Type":"ContainerStarted","Data":"ea9b893849036b211d240775d8b0da277ed3a369f0988302ba713a29fd192e60"} Nov 02 14:27:58.244606 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:58.244543 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl" Nov 02 14:27:58.293230 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:58.293177 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf" podStartSLOduration=1.293162387 podStartE2EDuration="1.293162387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:27:58.292514243 +0000 UTC m=+1843.476361579" watchObservedRunningTime="2026-11-02 14:27:58.293162387 +0000 UTC m=+1843.477009703" Nov 02 14:27:58.337359 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:27:58.337290 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl" podStartSLOduration=2.337274492 podStartE2EDuration="2.337274492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:27:58.334885872 +0000 UTC m=+1843.518733210" watchObservedRunningTime="2026-11-02 14:27:58.337274492 +0000 UTC m=+1843.521121828" Nov 02 14:28:01.035812 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.035748 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:28:01.183430 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.183396 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a1faca5-3a0e-4a93-afda-78b9aa799050-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1faca5-3a0e-4a93-afda-78b9aa799050-istiod-ca-cert\") pod \"9a1faca5-3a0e-4a93-afda-78b9aa799050\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " Nov 02 14:28:01.183430 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.183435 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1faca5-3a0e-4a93-afda-78b9aa799050-kube-api-access-797cr\" (UniqueName: \"kubernetes.io/projected/9a1faca5-3a0e-4a93-afda-78b9aa799050-kube-api-access-797cr\") pod \"9a1faca5-3a0e-4a93-afda-78b9aa799050\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " Nov 02 14:28:01.183683 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.183488 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-workload-socket\") pod \"9a1faca5-3a0e-4a93-afda-78b9aa799050\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " Nov 02 14:28:01.183683 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.183508 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-workload-certs\") pod \"9a1faca5-3a0e-4a93-afda-78b9aa799050\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " Nov 02 14:28:01.183683 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.183558 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-token\" (UniqueName: \"kubernetes.io/projected/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-token\") pod \"9a1faca5-3a0e-4a93-afda-78b9aa799050\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " Nov 02 14:28:01.183841 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.183784 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-podinfo\") pod \"9a1faca5-3a0e-4a93-afda-78b9aa799050\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " Nov 02 14:28:01.183892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.183844 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-envoy\") pod \"9a1faca5-3a0e-4a93-afda-78b9aa799050\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " Nov 02 14:28:01.183892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.183883 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-data\") pod \"9a1faca5-3a0e-4a93-afda-78b9aa799050\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " Nov 02 14:28:01.184031 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.183899 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-workload-certs" pod "9a1faca5-3a0e-4a93-afda-78b9aa799050" (UID: "9a1faca5-3a0e-4a93-afda-78b9aa799050"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:01.184031 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.183908 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-workload-socket" pod "9a1faca5-3a0e-4a93-afda-78b9aa799050" (UID: "9a1faca5-3a0e-4a93-afda-78b9aa799050"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:01.184031 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.183928 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-credential-socket\") pod \"9a1faca5-3a0e-4a93-afda-78b9aa799050\" (UID: \"9a1faca5-3a0e-4a93-afda-78b9aa799050\") " Nov 02 14:28:01.184031 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.183983 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1faca5-3a0e-4a93-afda-78b9aa799050-istiod-ca-cert" pod "9a1faca5-3a0e-4a93-afda-78b9aa799050" (UID: "9a1faca5-3a0e-4a93-afda-78b9aa799050"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:01.184216 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.184194 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-credential-socket" pod "9a1faca5-3a0e-4a93-afda-78b9aa799050" (UID: "9a1faca5-3a0e-4a93-afda-78b9aa799050"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:01.184250 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.184208 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-data" pod "9a1faca5-3a0e-4a93-afda-78b9aa799050" (UID: "9a1faca5-3a0e-4a93-afda-78b9aa799050"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:01.184573 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.184550 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:01.184672 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.184576 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:01.184672 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.184592 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1faca5-3a0e-4a93-afda-78b9aa799050-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:01.184672 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.184607 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:01.184672 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.184620 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:01.185924 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.185903 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1faca5-3a0e-4a93-afda-78b9aa799050-kube-api-access-797cr" pod "9a1faca5-3a0e-4a93-afda-78b9aa799050" (UID: "9a1faca5-3a0e-4a93-afda-78b9aa799050"). InnerVolumeSpecName "kube-api-access-797cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:01.185924 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.185913 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-token" pod "9a1faca5-3a0e-4a93-afda-78b9aa799050" (UID: "9a1faca5-3a0e-4a93-afda-78b9aa799050"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:01.186102 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.186086 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-podinfo" pod "9a1faca5-3a0e-4a93-afda-78b9aa799050" (UID: "9a1faca5-3a0e-4a93-afda-78b9aa799050"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:28:01.186145 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.186117 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-envoy" pod "9a1faca5-3a0e-4a93-afda-78b9aa799050" (UID: "9a1faca5-3a0e-4a93-afda-78b9aa799050"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:01.281445 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.281405 4866 generic.go:356] "Generic (PLEG): container finished" podID="9a1faca5-3a0e-4a93-afda-78b9aa799050" containerID="84ef9de0c7303f2610b22e64dc5d5c169d4839772c09628d0e4a45fabf089fdd" exitCode=0 Nov 02 14:28:01.281614 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.281463 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" Nov 02 14:28:01.281614 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.281484 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" event={"ID":"9a1faca5-3a0e-4a93-afda-78b9aa799050","Type":"ContainerDied","Data":"84ef9de0c7303f2610b22e64dc5d5c169d4839772c09628d0e4a45fabf089fdd"} Nov 02 14:28:01.281614 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.281570 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s" event={"ID":"9a1faca5-3a0e-4a93-afda-78b9aa799050","Type":"ContainerDied","Data":"e974320e3b2407a1242ef2eda1e4e0bdc778e44f7bf0dcf03a4e5db191a8777d"} Nov 02 14:28:01.281614 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.281593 4866 scope.go:122] "RemoveContainer" containerID="84ef9de0c7303f2610b22e64dc5d5c169d4839772c09628d0e4a45fabf089fdd" Nov 02 14:28:01.285343 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.285316 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:01.285343 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.285337 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:01.285343 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.285346 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1faca5-3a0e-4a93-afda-78b9aa799050-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:01.285575 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.285355 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-797cr\" (UniqueName: \"kubernetes.io/projected/9a1faca5-3a0e-4a93-afda-78b9aa799050-kube-api-access-797cr\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:01.292992 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.292941 4866 scope.go:122] "RemoveContainer" containerID="84ef9de0c7303f2610b22e64dc5d5c169d4839772c09628d0e4a45fabf089fdd" Nov 02 14:28:01.293331 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:28:01.293302 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ef9de0c7303f2610b22e64dc5d5c169d4839772c09628d0e4a45fabf089fdd\": container with ID starting with 84ef9de0c7303f2610b22e64dc5d5c169d4839772c09628d0e4a45fabf089fdd not found: ID does not exist" containerID="84ef9de0c7303f2610b22e64dc5d5c169d4839772c09628d0e4a45fabf089fdd" Nov 02 14:28:01.293389 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.293343 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ef9de0c7303f2610b22e64dc5d5c169d4839772c09628d0e4a45fabf089fdd"} err="failed to get container status \"84ef9de0c7303f2610b22e64dc5d5c169d4839772c09628d0e4a45fabf089fdd\": rpc error: code = NotFound desc = could not find container \"84ef9de0c7303f2610b22e64dc5d5c169d4839772c09628d0e4a45fabf089fdd\": container with ID starting with 84ef9de0c7303f2610b22e64dc5d5c169d4839772c09628d0e4a45fabf089fdd not found: ID does not exist" Nov 02 14:28:01.338599 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.338506 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s"] Nov 02 14:28:01.343699 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:01.343667 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lk65s"] Nov 02 14:28:02.176340 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.176292 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v"] Nov 02 14:28:02.176902 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.176888 4866 cpu_manager.go:414] "containerMap: removing container" podUID="9a1faca5-3a0e-4a93-afda-78b9aa799050" containerName="istio-proxy" Nov 02 14:28:02.176963 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.176903 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a1faca5-3a0e-4a93-afda-78b9aa799050" containerName="istio-proxy" Nov 02 14:28:02.177060 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.177050 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a1faca5-3a0e-4a93-afda-78b9aa799050" containerName="istio-proxy" Nov 02 14:28:02.181794 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.181776 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" Nov 02 14:28:02.205280 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.205251 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v"] Nov 02 14:28:02.239724 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.239699 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" Nov 02 14:28:02.298495 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.298456 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v\" (UID: \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" Nov 02 14:28:02.298652 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.298504 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v\" (UID: \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" Nov 02 14:28:02.298715 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.298678 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v\" (UID: \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" Nov 02 14:28:02.298715 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.298709 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hhcg\" (UniqueName: \"kubernetes.io/projected/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-kube-api-access-2hhcg\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v\" (UID: \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" Nov 02 14:28:02.399782 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.399741 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v\" (UID: \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" Nov 02 14:28:02.399782 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.399786 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hhcg\" (UniqueName: \"kubernetes.io/projected/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-kube-api-access-2hhcg\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v\" (UID: \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" Nov 02 14:28:02.400080 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.399855 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v\" (UID: \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" Nov 02 14:28:02.400080 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.400029 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v\" (UID: \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" Nov 02 14:28:02.400670 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.400643 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v\" (UID: \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" Nov 02 14:28:02.400670 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.400661 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v\" (UID: \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" Nov 02 14:28:02.402451 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.402430 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v\" (UID: \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" Nov 02 14:28:02.424281 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.424250 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hhcg\" (UniqueName: \"kubernetes.io/projected/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-kube-api-access-2hhcg\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v\" (UID: \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" Nov 02 14:28:02.495562 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.495529 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" Nov 02 14:28:02.803755 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.803726 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v"] Nov 02 14:28:02.805720 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:28:02.805690 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266eae4f_b995_46e0_b8b6_ad6dcfbd6e5e.slice/crio-d2bd1bbe022abef4631300f4787dd029dec0abf85eedf091f26519a4922ba61c WatchSource:0}: Error finding container d2bd1bbe022abef4631300f4787dd029dec0abf85eedf091f26519a4922ba61c: Status 404 returned error can't find the container with id d2bd1bbe022abef4631300f4787dd029dec0abf85eedf091f26519a4922ba61c Nov 02 14:28:02.945905 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:02.945870 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a1faca5-3a0e-4a93-afda-78b9aa799050" path="/var/lib/kubelet/pods/9a1faca5-3a0e-4a93-afda-78b9aa799050/volumes" Nov 02 14:28:03.296219 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:03.296185 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" event={"ID":"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e","Type":"ContainerStarted","Data":"c62250b6f1ddac66e833331a9d65ce9fcdaa5d3e0f172f900c4e39521746c55e"} Nov 02 14:28:03.296589 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:03.296241 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" event={"ID":"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e","Type":"ContainerStarted","Data":"d2bd1bbe022abef4631300f4787dd029dec0abf85eedf091f26519a4922ba61c"} Nov 02 14:28:03.296589 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:03.296324 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" Nov 02 14:28:03.343576 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:03.343529 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" podStartSLOduration=1.343514185 podStartE2EDuration="1.343514185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:03.340030355 +0000 UTC m=+1848.523877692" watchObservedRunningTime="2026-11-02 14:28:03.343514185 +0000 UTC m=+1848.527361522" Nov 02 14:28:06.009904 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:28:06.009865 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:28:09.262572 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:09.262535 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf" Nov 02 14:28:09.263081 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:09.262975 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl" Nov 02 14:28:14.317160 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:14.317129 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" Nov 02 14:28:14.392567 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:14.392538 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm"] Nov 02 14:28:19.393234 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:19.393149 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" podUID="4722986e-2218-4c2d-aac6-3b4a346e81ef" containerName="payload-processing" containerID="cri-o://bc6e13f8ee7effda3cf80ba92f54a9314c3454f4f7f3a1ee3d453608050f0650" gracePeriod=25 Nov 02 14:28:19.662376 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:19.662316 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" Nov 02 14:28:19.821477 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:19.821448 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4722986e-2218-4c2d-aac6-3b4a346e81ef-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4722986e-2218-4c2d-aac6-3b4a346e81ef-routing-overlay-volume\") pod \"4722986e-2218-4c2d-aac6-3b4a346e81ef\" (UID: \"4722986e-2218-4c2d-aac6-3b4a346e81ef\") " Nov 02 14:28:19.821675 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:19.821514 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4722986e-2218-4c2d-aac6-3b4a346e81ef-kube-api-access-288cl\" (UniqueName: \"kubernetes.io/projected/4722986e-2218-4c2d-aac6-3b4a346e81ef-kube-api-access-288cl\") pod \"4722986e-2218-4c2d-aac6-3b4a346e81ef\" (UID: \"4722986e-2218-4c2d-aac6-3b4a346e81ef\") " Nov 02 14:28:19.821675 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:19.821600 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4722986e-2218-4c2d-aac6-3b4a346e81ef-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4722986e-2218-4c2d-aac6-3b4a346e81ef-provider-credentials\") pod \"4722986e-2218-4c2d-aac6-3b4a346e81ef\" (UID: \"4722986e-2218-4c2d-aac6-3b4a346e81ef\") " Nov 02 14:28:19.821806 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:19.821691 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4722986e-2218-4c2d-aac6-3b4a346e81ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4722986e-2218-4c2d-aac6-3b4a346e81ef-plugins-config-volume\") pod \"4722986e-2218-4c2d-aac6-3b4a346e81ef\" (UID: \"4722986e-2218-4c2d-aac6-3b4a346e81ef\") " Nov 02 14:28:19.821890 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:19.821863 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4722986e-2218-4c2d-aac6-3b4a346e81ef-routing-overlay-volume" pod "4722986e-2218-4c2d-aac6-3b4a346e81ef" (UID: "4722986e-2218-4c2d-aac6-3b4a346e81ef"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:19.822123 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:19.822100 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4722986e-2218-4c2d-aac6-3b4a346e81ef-plugins-config-volume" pod "4722986e-2218-4c2d-aac6-3b4a346e81ef" (UID: "4722986e-2218-4c2d-aac6-3b4a346e81ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:19.822217 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:19.822196 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4722986e-2218-4c2d-aac6-3b4a346e81ef-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:19.822217 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:19.822220 4866 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4722986e-2218-4c2d-aac6-3b4a346e81ef-routing-overlay-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:19.823740 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:19.823715 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4722986e-2218-4c2d-aac6-3b4a346e81ef-kube-api-access-288cl" pod "4722986e-2218-4c2d-aac6-3b4a346e81ef" (UID: "4722986e-2218-4c2d-aac6-3b4a346e81ef"). InnerVolumeSpecName "kube-api-access-288cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:19.823740 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:19.823731 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4722986e-2218-4c2d-aac6-3b4a346e81ef-provider-credentials" pod "4722986e-2218-4c2d-aac6-3b4a346e81ef" (UID: "4722986e-2218-4c2d-aac6-3b4a346e81ef"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:19.923793 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:19.923711 4866 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4722986e-2218-4c2d-aac6-3b4a346e81ef-provider-credentials\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:19.923793 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:19.923742 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-288cl\" (UniqueName: \"kubernetes.io/projected/4722986e-2218-4c2d-aac6-3b4a346e81ef-kube-api-access-288cl\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:20.423113 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.423073 4866 generic.go:356] "Generic (PLEG): container finished" podID="4722986e-2218-4c2d-aac6-3b4a346e81ef" containerID="bc6e13f8ee7effda3cf80ba92f54a9314c3454f4f7f3a1ee3d453608050f0650" exitCode=0 Nov 02 14:28:20.423599 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.423120 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" event={"ID":"4722986e-2218-4c2d-aac6-3b4a346e81ef","Type":"ContainerDied","Data":"bc6e13f8ee7effda3cf80ba92f54a9314c3454f4f7f3a1ee3d453608050f0650"} Nov 02 14:28:20.423599 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.423174 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" Nov 02 14:28:20.423599 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.423191 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm" event={"ID":"4722986e-2218-4c2d-aac6-3b4a346e81ef","Type":"ContainerDied","Data":"228be522a8e5e95373941887df99f7a972adc16c8391c105ec67e25d1ce9c9fa"} Nov 02 14:28:20.423599 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.423219 4866 scope.go:122] "RemoveContainer" containerID="bc6e13f8ee7effda3cf80ba92f54a9314c3454f4f7f3a1ee3d453608050f0650" Nov 02 14:28:20.434688 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.434670 4866 scope.go:122] "RemoveContainer" containerID="bc6e13f8ee7effda3cf80ba92f54a9314c3454f4f7f3a1ee3d453608050f0650" Nov 02 14:28:20.435014 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:28:20.434965 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc6e13f8ee7effda3cf80ba92f54a9314c3454f4f7f3a1ee3d453608050f0650\": container with ID starting with bc6e13f8ee7effda3cf80ba92f54a9314c3454f4f7f3a1ee3d453608050f0650 not found: ID does not exist" containerID="bc6e13f8ee7effda3cf80ba92f54a9314c3454f4f7f3a1ee3d453608050f0650" Nov 02 14:28:20.435014 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.434993 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc6e13f8ee7effda3cf80ba92f54a9314c3454f4f7f3a1ee3d453608050f0650"} err="failed to get container status \"bc6e13f8ee7effda3cf80ba92f54a9314c3454f4f7f3a1ee3d453608050f0650\": rpc error: code = NotFound desc = could not find container \"bc6e13f8ee7effda3cf80ba92f54a9314c3454f4f7f3a1ee3d453608050f0650\": container with ID starting with bc6e13f8ee7effda3cf80ba92f54a9314c3454f4f7f3a1ee3d453608050f0650 not found: ID does not exist" Nov 02 14:28:20.475002 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.474973 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm"] Nov 02 14:28:20.486524 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.486495 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-84bcs8zkm"] Nov 02 14:28:20.593392 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.593357 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb"] Nov 02 14:28:20.593919 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.593906 4866 cpu_manager.go:414] "containerMap: removing container" podUID="4722986e-2218-4c2d-aac6-3b4a346e81ef" containerName="payload-processing" Nov 02 14:28:20.593997 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.593920 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4722986e-2218-4c2d-aac6-3b4a346e81ef" containerName="payload-processing" Nov 02 14:28:20.594097 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.594086 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="4722986e-2218-4c2d-aac6-3b4a346e81ef" containerName="payload-processing" Nov 02 14:28:20.598897 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.598876 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" Nov 02 14:28:20.618385 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.618349 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-5a4525-serving-cert\"" Nov 02 14:28:20.618759 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.618737 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb"] Nov 02 14:28:20.733236 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.733204 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9msn2\" (UniqueName: \"kubernetes.io/projected/7697c9b1-291e-4bed-aa54-7e21c05677bd-kube-api-access-9msn2\") pod \"maas-api-e2e-explicit-5a4525-68869c56dd-jltfb\" (UID: \"7697c9b1-291e-4bed-aa54-7e21c05677bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" Nov 02 14:28:20.733421 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.733264 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7697c9b1-291e-4bed-aa54-7e21c05677bd-maas-api-tls\") pod \"maas-api-e2e-explicit-5a4525-68869c56dd-jltfb\" (UID: \"7697c9b1-291e-4bed-aa54-7e21c05677bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" Nov 02 14:28:20.733421 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.733343 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7697c9b1-291e-4bed-aa54-7e21c05677bd-metrics-certs\") pod \"maas-api-e2e-explicit-5a4525-68869c56dd-jltfb\" (UID: \"7697c9b1-291e-4bed-aa54-7e21c05677bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" Nov 02 14:28:20.733421 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.733401 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7697c9b1-291e-4bed-aa54-7e21c05677bd-openshift-service-ca\") pod \"maas-api-e2e-explicit-5a4525-68869c56dd-jltfb\" (UID: \"7697c9b1-291e-4bed-aa54-7e21c05677bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" Nov 02 14:28:20.838889 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.838836 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7697c9b1-291e-4bed-aa54-7e21c05677bd-maas-api-tls\") pod \"maas-api-e2e-explicit-5a4525-68869c56dd-jltfb\" (UID: \"7697c9b1-291e-4bed-aa54-7e21c05677bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" Nov 02 14:28:20.839121 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.838907 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7697c9b1-291e-4bed-aa54-7e21c05677bd-metrics-certs\") pod \"maas-api-e2e-explicit-5a4525-68869c56dd-jltfb\" (UID: \"7697c9b1-291e-4bed-aa54-7e21c05677bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" Nov 02 14:28:20.839121 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.838982 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7697c9b1-291e-4bed-aa54-7e21c05677bd-openshift-service-ca\") pod \"maas-api-e2e-explicit-5a4525-68869c56dd-jltfb\" (UID: \"7697c9b1-291e-4bed-aa54-7e21c05677bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" Nov 02 14:28:20.839228 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.839216 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9msn2\" (UniqueName: \"kubernetes.io/projected/7697c9b1-291e-4bed-aa54-7e21c05677bd-kube-api-access-9msn2\") pod \"maas-api-e2e-explicit-5a4525-68869c56dd-jltfb\" (UID: \"7697c9b1-291e-4bed-aa54-7e21c05677bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" Nov 02 14:28:20.844443 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.844408 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7697c9b1-291e-4bed-aa54-7e21c05677bd-openshift-service-ca\") pod \"maas-api-e2e-explicit-5a4525-68869c56dd-jltfb\" (UID: \"7697c9b1-291e-4bed-aa54-7e21c05677bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" Nov 02 14:28:20.847146 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.847117 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7697c9b1-291e-4bed-aa54-7e21c05677bd-metrics-certs\") pod \"maas-api-e2e-explicit-5a4525-68869c56dd-jltfb\" (UID: \"7697c9b1-291e-4bed-aa54-7e21c05677bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" Nov 02 14:28:20.849856 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.849826 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7697c9b1-291e-4bed-aa54-7e21c05677bd-maas-api-tls\") pod \"maas-api-e2e-explicit-5a4525-68869c56dd-jltfb\" (UID: \"7697c9b1-291e-4bed-aa54-7e21c05677bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" Nov 02 14:28:20.863615 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.863584 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9msn2\" (UniqueName: \"kubernetes.io/projected/7697c9b1-291e-4bed-aa54-7e21c05677bd-kube-api-access-9msn2\") pod \"maas-api-e2e-explicit-5a4525-68869c56dd-jltfb\" (UID: \"7697c9b1-291e-4bed-aa54-7e21c05677bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" Nov 02 14:28:20.912733 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.912696 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" Nov 02 14:28:20.950680 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:20.950630 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4722986e-2218-4c2d-aac6-3b4a346e81ef" path="/var/lib/kubelet/pods/4722986e-2218-4c2d-aac6-3b4a346e81ef/volumes" Nov 02 14:28:21.227913 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:21.227883 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb"] Nov 02 14:28:21.228819 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:28:21.228791 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7697c9b1_291e_4bed_aa54_7e21c05677bd.slice/crio-0707af7c81319d91a4b0d77442b86444edced17e843d4b8017d87d642954cbd3 WatchSource:0}: Error finding container 0707af7c81319d91a4b0d77442b86444edced17e843d4b8017d87d642954cbd3: Status 404 returned error can't find the container with id 0707af7c81319d91a4b0d77442b86444edced17e843d4b8017d87d642954cbd3 Nov 02 14:28:21.430318 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:21.430209 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" event={"ID":"7697c9b1-291e-4bed-aa54-7e21c05677bd","Type":"ContainerStarted","Data":"0707af7c81319d91a4b0d77442b86444edced17e843d4b8017d87d642954cbd3"} Nov 02 14:28:22.438285 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:22.438233 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" event={"ID":"7697c9b1-291e-4bed-aa54-7e21c05677bd","Type":"ContainerStarted","Data":"c9eb8d3fa72f5dca5b6b5c941cff2c0bcb40853f3b6215671ea69a50e63aece9"} Nov 02 14:28:22.438699 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:22.438327 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" Nov 02 14:28:22.485107 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:22.485045 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" podStartSLOduration=2.038295221 podStartE2EDuration="2.485026138s" totalImagesPullingTime="446.730917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:22.480969517 +0000 UTC m=+1867.664816844" watchObservedRunningTime="2026-11-02 14:28:22.485026138 +0000 UTC m=+1867.668873479" Nov 02 14:28:26.079888 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:26.079852 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m"] Nov 02 14:28:26.080378 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:26.080112 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" podUID="8d0a11e0-7de7-4661-aac7-f785b2dc7777" containerName="istio-proxy" containerID="cri-o://e76a519d7b570438709f49cc02af0b25e971e090bf7d5842f897a6e8d291a2ce" gracePeriod=30 Nov 02 14:28:28.449540 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:28.449511 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" Nov 02 14:28:31.374603 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.374029 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:28:31.511207 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.511163 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-podinfo\") pod \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.514060 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-credential-socket\") pod \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.514153 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-workload-certs\") pod \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.514196 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istiod-ca-cert\") pod \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.514281 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-token\" (UniqueName: \"kubernetes.io/projected/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-token\") pod \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.514312 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-envoy\") pod \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.514377 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-workload-socket\") pod \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.514436 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d0a11e0-7de7-4661-aac7-f785b2dc7777-kube-api-access-875bs\" (UniqueName: \"kubernetes.io/projected/8d0a11e0-7de7-4661-aac7-f785b2dc7777-kube-api-access-875bs\") pod \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.514585 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-data\") pod \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\" (UID: \"8d0a11e0-7de7-4661-aac7-f785b2dc7777\") " Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.515294 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-data" pod "8d0a11e0-7de7-4661-aac7-f785b2dc7777" (UID: "8d0a11e0-7de7-4661-aac7-f785b2dc7777"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.515724 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-workload-certs" pod "8d0a11e0-7de7-4661-aac7-f785b2dc7777" (UID: "8d0a11e0-7de7-4661-aac7-f785b2dc7777"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.515745 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-workload-socket" pod "8d0a11e0-7de7-4661-aac7-f785b2dc7777" (UID: "8d0a11e0-7de7-4661-aac7-f785b2dc7777"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.515821 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istiod-ca-cert" pod "8d0a11e0-7de7-4661-aac7-f785b2dc7777" (UID: "8d0a11e0-7de7-4661-aac7-f785b2dc7777"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.516102 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-credential-socket" pod "8d0a11e0-7de7-4661-aac7-f785b2dc7777" (UID: "8d0a11e0-7de7-4661-aac7-f785b2dc7777"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.517188 4866 generic.go:356] "Generic (PLEG): container finished" podID="8d0a11e0-7de7-4661-aac7-f785b2dc7777" containerID="e76a519d7b570438709f49cc02af0b25e971e090bf7d5842f897a6e8d291a2ce" exitCode=0 Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.517236 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.517263 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" event={"ID":"8d0a11e0-7de7-4661-aac7-f785b2dc7777","Type":"ContainerDied","Data":"e76a519d7b570438709f49cc02af0b25e971e090bf7d5842f897a6e8d291a2ce"} Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.517317 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m" event={"ID":"8d0a11e0-7de7-4661-aac7-f785b2dc7777","Type":"ContainerDied","Data":"8c7d0e70aa4df5880d7020d7d9f0b71474d11aec5dad25265d9cc4908b6f80c6"} Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.517347 4866 scope.go:122] "RemoveContainer" containerID="e76a519d7b570438709f49cc02af0b25e971e090bf7d5842f897a6e8d291a2ce" Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.517588 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-token" pod "8d0a11e0-7de7-4661-aac7-f785b2dc7777" (UID: "8d0a11e0-7de7-4661-aac7-f785b2dc7777"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.517824 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-podinfo" pod "8d0a11e0-7de7-4661-aac7-f785b2dc7777" (UID: "8d0a11e0-7de7-4661-aac7-f785b2dc7777"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.518189 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0a11e0-7de7-4661-aac7-f785b2dc7777-kube-api-access-875bs" pod "8d0a11e0-7de7-4661-aac7-f785b2dc7777" (UID: "8d0a11e0-7de7-4661-aac7-f785b2dc7777"). InnerVolumeSpecName "kube-api-access-875bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:31.523210 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.518251 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-envoy" pod "8d0a11e0-7de7-4661-aac7-f785b2dc7777" (UID: "8d0a11e0-7de7-4661-aac7-f785b2dc7777"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:31.543523 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.543500 4866 scope.go:122] "RemoveContainer" containerID="e76a519d7b570438709f49cc02af0b25e971e090bf7d5842f897a6e8d291a2ce" Nov 02 14:28:31.543856 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:28:31.543825 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e76a519d7b570438709f49cc02af0b25e971e090bf7d5842f897a6e8d291a2ce\": container with ID starting with e76a519d7b570438709f49cc02af0b25e971e090bf7d5842f897a6e8d291a2ce not found: ID does not exist" containerID="e76a519d7b570438709f49cc02af0b25e971e090bf7d5842f897a6e8d291a2ce" Nov 02 14:28:31.543904 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.543867 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76a519d7b570438709f49cc02af0b25e971e090bf7d5842f897a6e8d291a2ce"} err="failed to get container status \"e76a519d7b570438709f49cc02af0b25e971e090bf7d5842f897a6e8d291a2ce\": rpc error: code = NotFound desc = could not find container \"e76a519d7b570438709f49cc02af0b25e971e090bf7d5842f897a6e8d291a2ce\": container with ID starting with e76a519d7b570438709f49cc02af0b25e971e090bf7d5842f897a6e8d291a2ce not found: ID does not exist" Nov 02 14:28:31.616131 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.616100 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:31.616131 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.616128 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:31.616131 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.616137 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:31.616366 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.616146 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:31.616366 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.616155 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:31.616366 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.616162 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:31.616366 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.616170 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:31.616366 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.616177 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a11e0-7de7-4661-aac7-f785b2dc7777-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:31.616366 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.616186 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-875bs\" (UniqueName: \"kubernetes.io/projected/8d0a11e0-7de7-4661-aac7-f785b2dc7777-kube-api-access-875bs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:31.864340 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.864300 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m"] Nov 02 14:28:31.876356 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:31.876322 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-9dc2m"] Nov 02 14:28:32.947433 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:32.947398 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d0a11e0-7de7-4661-aac7-f785b2dc7777" path="/var/lib/kubelet/pods/8d0a11e0-7de7-4661-aac7-f785b2dc7777/volumes" Nov 02 14:28:36.032126 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:28:36.031536 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:28:36.087339 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:36.087308 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v"] Nov 02 14:28:41.087824 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.087747 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" podUID="266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e" containerName="payload-processing" containerID="cri-o://c62250b6f1ddac66e833331a9d65ce9fcdaa5d3e0f172f900c4e39521746c55e" gracePeriod=25 Nov 02 14:28:41.373989 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.373959 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" Nov 02 14:28:41.465159 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.465127 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-provider-credentials\") pod \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\" (UID: \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\") " Nov 02 14:28:41.465350 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.465177 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-routing-overlay-volume\") pod \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\" (UID: \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\") " Nov 02 14:28:41.465350 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.465230 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-kube-api-access-2hhcg\" (UniqueName: \"kubernetes.io/projected/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-kube-api-access-2hhcg\") pod \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\" (UID: \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\") " Nov 02 14:28:41.465350 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.465276 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-plugins-config-volume\") pod \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\" (UID: \"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e\") " Nov 02 14:28:41.465698 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.465674 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-routing-overlay-volume" pod "266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e" (UID: "266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:41.465795 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.465776 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-plugins-config-volume" pod "266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e" (UID: "266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:41.467454 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.467428 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-provider-credentials" pod "266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e" (UID: "266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:41.467559 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.467519 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-kube-api-access-2hhcg" pod "266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e" (UID: "266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e"). InnerVolumeSpecName "kube-api-access-2hhcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:41.567064 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.567031 4866 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-provider-credentials\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:41.567064 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.567062 4866 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-routing-overlay-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:41.567264 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.567075 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hhcg\" (UniqueName: \"kubernetes.io/projected/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-kube-api-access-2hhcg\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:41.567264 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.567088 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:28:41.593222 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.593182 4866 generic.go:356] "Generic (PLEG): container finished" podID="266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e" containerID="c62250b6f1ddac66e833331a9d65ce9fcdaa5d3e0f172f900c4e39521746c55e" exitCode=0 Nov 02 14:28:41.593404 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.593237 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" Nov 02 14:28:41.593404 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.593271 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" event={"ID":"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e","Type":"ContainerDied","Data":"c62250b6f1ddac66e833331a9d65ce9fcdaa5d3e0f172f900c4e39521746c55e"} Nov 02 14:28:41.593404 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.593342 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v" event={"ID":"266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e","Type":"ContainerDied","Data":"d2bd1bbe022abef4631300f4787dd029dec0abf85eedf091f26519a4922ba61c"} Nov 02 14:28:41.593404 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.593374 4866 scope.go:122] "RemoveContainer" containerID="c62250b6f1ddac66e833331a9d65ce9fcdaa5d3e0f172f900c4e39521746c55e" Nov 02 14:28:41.604585 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.604561 4866 scope.go:122] "RemoveContainer" containerID="c62250b6f1ddac66e833331a9d65ce9fcdaa5d3e0f172f900c4e39521746c55e" Nov 02 14:28:41.604898 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:28:41.604879 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c62250b6f1ddac66e833331a9d65ce9fcdaa5d3e0f172f900c4e39521746c55e\": container with ID starting with c62250b6f1ddac66e833331a9d65ce9fcdaa5d3e0f172f900c4e39521746c55e not found: ID does not exist" containerID="c62250b6f1ddac66e833331a9d65ce9fcdaa5d3e0f172f900c4e39521746c55e" Nov 02 14:28:41.604982 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.604906 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c62250b6f1ddac66e833331a9d65ce9fcdaa5d3e0f172f900c4e39521746c55e"} err="failed to get container status \"c62250b6f1ddac66e833331a9d65ce9fcdaa5d3e0f172f900c4e39521746c55e\": rpc error: code = NotFound desc = could not find container \"c62250b6f1ddac66e833331a9d65ce9fcdaa5d3e0f172f900c4e39521746c55e\": container with ID starting with c62250b6f1ddac66e833331a9d65ce9fcdaa5d3e0f172f900c4e39521746c55e not found: ID does not exist" Nov 02 14:28:41.646384 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.646356 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v"] Nov 02 14:28:41.650737 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:41.650709 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-5cc5qhm8v"] Nov 02 14:28:42.601286 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:42.601259 4866 generic.go:356] "Generic (PLEG): container finished" podID="c91b5d92-2a59-46c7-ac62-f398a392a8ff" containerID="d63f03129bbf648a49d5216dbedb2e3213b6348960d81522142b0b3b31842d0b" exitCode=0 Nov 02 14:28:42.601713 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:42.601322 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" event={"ID":"c91b5d92-2a59-46c7-ac62-f398a392a8ff","Type":"ContainerDied","Data":"d63f03129bbf648a49d5216dbedb2e3213b6348960d81522142b0b3b31842d0b"} Nov 02 14:28:42.601754 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:42.601711 4866 scope.go:122] "RemoveContainer" containerID="d63f03129bbf648a49d5216dbedb2e3213b6348960d81522142b0b3b31842d0b" Nov 02 14:28:42.603296 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:42.603269 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:28:42.603375 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:42.603342 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:28:42.603415 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:42.603377 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:28:42.954976 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:42.950970 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e" path="/var/lib/kubelet/pods/266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e/volumes" Nov 02 14:28:43.632383 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:43.632342 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" event={"ID":"c91b5d92-2a59-46c7-ac62-f398a392a8ff","Type":"ContainerStarted","Data":"f1e87085bce0956eca1d982923b8b4001babf71e6ac41dcefe1143e0242f8350"} Nov 02 14:28:43.724233 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:43.724196 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:28:43.724476 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:43.724446 4866 patch_prober.go:29] interesting pod/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 14:28:43.724557 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:43.724511 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" podUID="c91b5d92-2a59-46c7-ac62-f398a392a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 14:28:44.709971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:44.703449 4866 patch_prober.go:29] interesting pod/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 14:28:44.709971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:44.703544 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" podUID="c91b5d92-2a59-46c7-ac62-f398a392a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 14:28:45.722992 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:45.719398 4866 patch_prober.go:29] interesting pod/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 14:28:45.723571 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:45.723170 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" podUID="c91b5d92-2a59-46c7-ac62-f398a392a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 14:28:46.719409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:46.719365 4866 patch_prober.go:29] interesting pod/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 14:28:46.719721 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:46.719685 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" podUID="c91b5d92-2a59-46c7-ac62-f398a392a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 14:28:47.712377 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:47.712340 4866 patch_prober.go:29] interesting pod/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 14:28:47.713050 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:47.713005 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" podUID="c91b5d92-2a59-46c7-ac62-f398a392a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 14:28:48.703306 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:48.703268 4866 patch_prober.go:29] interesting pod/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 14:28:48.703489 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:48.703352 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" podUID="c91b5d92-2a59-46c7-ac62-f398a392a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 14:28:49.707499 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:49.707470 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:28:49.707886 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:49.707750 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:28:49.708427 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:28:49.708412 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:29:03.040162 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.040121 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx"] Nov 02 14:29:03.040717 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.040701 4866 cpu_manager.go:414] "containerMap: removing container" podUID="8d0a11e0-7de7-4661-aac7-f785b2dc7777" containerName="istio-proxy" Nov 02 14:29:03.040764 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.040718 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d0a11e0-7de7-4661-aac7-f785b2dc7777" containerName="istio-proxy" Nov 02 14:29:03.040764 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.040735 4866 cpu_manager.go:414] "containerMap: removing container" podUID="266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e" containerName="payload-processing" Nov 02 14:29:03.040764 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.040741 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e" containerName="payload-processing" Nov 02 14:29:03.040857 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.040824 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d0a11e0-7de7-4661-aac7-f785b2dc7777" containerName="istio-proxy" Nov 02 14:29:03.040857 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.040835 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="266eae4f-b995-46e0-b8b6-ad6dcfbd6e5e" containerName="payload-processing" Nov 02 14:29:03.045602 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.045580 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx" Nov 02 14:29:03.063422 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.063392 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d0c9b9\"" Nov 02 14:29:03.073527 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.073487 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d0c9b9-dockercfg-wzrq9\"" Nov 02 14:29:03.120092 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.120050 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx"] Nov 02 14:29:03.132989 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.132903 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx7nx\" (UniqueName: \"kubernetes.io/projected/41e0f03e-11af-465d-aef2-8c3ef52a06bf-kube-api-access-vx7nx\") pod \"payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx\" (UID: \"41e0f03e-11af-465d-aef2-8c3ef52a06bf\") " pod="openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx" Nov 02 14:29:03.140984 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.139105 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e0f03e-11af-465d-aef2-8c3ef52a06bf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx\" (UID: \"41e0f03e-11af-465d-aef2-8c3ef52a06bf\") " pod="openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx" Nov 02 14:29:03.240763 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.240723 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e0f03e-11af-465d-aef2-8c3ef52a06bf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx\" (UID: \"41e0f03e-11af-465d-aef2-8c3ef52a06bf\") " pod="openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx" Nov 02 14:29:03.240993 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.240970 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx7nx\" (UniqueName: \"kubernetes.io/projected/41e0f03e-11af-465d-aef2-8c3ef52a06bf-kube-api-access-vx7nx\") pod \"payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx\" (UID: \"41e0f03e-11af-465d-aef2-8c3ef52a06bf\") " pod="openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx" Nov 02 14:29:03.241541 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.241523 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e0f03e-11af-465d-aef2-8c3ef52a06bf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx\" (UID: \"41e0f03e-11af-465d-aef2-8c3ef52a06bf\") " pod="openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx" Nov 02 14:29:03.279620 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.279578 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx7nx\" (UniqueName: \"kubernetes.io/projected/41e0f03e-11af-465d-aef2-8c3ef52a06bf-kube-api-access-vx7nx\") pod \"payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx\" (UID: \"41e0f03e-11af-465d-aef2-8c3ef52a06bf\") " pod="openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx" Nov 02 14:29:03.444743 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.444706 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx" Nov 02 14:29:03.757514 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.757483 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx"] Nov 02 14:29:03.758792 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:29:03.758763 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e0f03e_11af_465d_aef2_8c3ef52a06bf.slice/crio-3356a772a4e9e8472b93130f0fc5f753db7b921483a949a8a5da0f08283196cf WatchSource:0}: Error finding container 3356a772a4e9e8472b93130f0fc5f753db7b921483a949a8a5da0f08283196cf: Status 404 returned error can't find the container with id 3356a772a4e9e8472b93130f0fc5f753db7b921483a949a8a5da0f08283196cf Nov 02 14:29:03.823597 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:03.823568 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx" event={"ID":"41e0f03e-11af-465d-aef2-8c3ef52a06bf","Type":"ContainerStarted","Data":"3356a772a4e9e8472b93130f0fc5f753db7b921483a949a8a5da0f08283196cf"} Nov 02 14:29:04.831045 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:04.831006 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx" event={"ID":"41e0f03e-11af-465d-aef2-8c3ef52a06bf","Type":"ContainerStarted","Data":"df53f7a442a806c5aa847f8f7042db5fa3e1799fff160df74c1ab26cf9f70c87"} Nov 02 14:29:04.831547 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:04.831143 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx" Nov 02 14:29:04.873329 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:04.873270 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx" podStartSLOduration=1.8732558849999998 podStartE2EDuration="1.873255885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:29:04.872005167 +0000 UTC m=+1910.055852526" watchObservedRunningTime="2026-11-02 14:29:04.873255885 +0000 UTC m=+1910.057103221" Nov 02 14:29:06.010612 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:06.010568 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:29:08.247896 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.247862 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t"] Nov 02 14:29:08.252083 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.252061 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" Nov 02 14:29:08.284998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.283390 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"routing-overlay\"" Nov 02 14:29:08.284998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.283397 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"openshift-service-ca.crt\"" Nov 02 14:29:08.284998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.283431 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"kube-root-ca.crt\"" Nov 02 14:29:08.284998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.283464 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"payload-processing-external-model-plugins-e2e-extmod-a-944506\"" Nov 02 14:29:08.287064 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.287036 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t"] Nov 02 14:29:08.312088 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.312043 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"e2e-cfl-622143-provider-cred\"" Nov 02 14:29:08.312348 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.312326 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"payload-processing-external-model-e2e--4e96fc15-dockercfg-gffbp\"" Nov 02 14:29:08.318641 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.318611 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t\" (UID: \"3648aa77-bed0-4906-8f84-e11dceb7bf37\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" Nov 02 14:29:08.318795 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.318653 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3648aa77-bed0-4906-8f84-e11dceb7bf37-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t\" (UID: \"3648aa77-bed0-4906-8f84-e11dceb7bf37\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" Nov 02 14:29:08.318876 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.318855 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3648aa77-bed0-4906-8f84-e11dceb7bf37-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t\" (UID: \"3648aa77-bed0-4906-8f84-e11dceb7bf37\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" Nov 02 14:29:08.318913 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.318896 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhgmn\" (UniqueName: \"kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-kube-api-access-dhgmn\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t\" (UID: \"3648aa77-bed0-4906-8f84-e11dceb7bf37\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" Nov 02 14:29:08.420194 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.420150 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3648aa77-bed0-4906-8f84-e11dceb7bf37-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t\" (UID: \"3648aa77-bed0-4906-8f84-e11dceb7bf37\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" Nov 02 14:29:08.420386 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.420203 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhgmn\" (UniqueName: \"kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-kube-api-access-dhgmn\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t\" (UID: \"3648aa77-bed0-4906-8f84-e11dceb7bf37\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" Nov 02 14:29:08.420386 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.420247 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t\" (UID: \"3648aa77-bed0-4906-8f84-e11dceb7bf37\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" Nov 02 14:29:08.420386 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.420272 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3648aa77-bed0-4906-8f84-e11dceb7bf37-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t\" (UID: \"3648aa77-bed0-4906-8f84-e11dceb7bf37\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" Nov 02 14:29:08.420989 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.420923 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3648aa77-bed0-4906-8f84-e11dceb7bf37-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t\" (UID: \"3648aa77-bed0-4906-8f84-e11dceb7bf37\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" Nov 02 14:29:08.421173 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.421149 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3648aa77-bed0-4906-8f84-e11dceb7bf37-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t\" (UID: \"3648aa77-bed0-4906-8f84-e11dceb7bf37\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" Nov 02 14:29:08.422724 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.422699 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t\" (UID: \"3648aa77-bed0-4906-8f84-e11dceb7bf37\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" Nov 02 14:29:08.445079 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.445046 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhgmn\" (UniqueName: \"kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-kube-api-access-dhgmn\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t\" (UID: \"3648aa77-bed0-4906-8f84-e11dceb7bf37\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" Nov 02 14:29:08.565666 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.565571 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" Nov 02 14:29:08.875361 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.875172 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t"] Nov 02 14:29:08.875496 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:29:08.875394 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3648aa77_bed0_4906_8f84_e11dceb7bf37.slice/crio-45c6216a75f29d6e13098553e91f02193d51e40ecef93b80e2dbb1b9c14529ad WatchSource:0}: Error finding container 45c6216a75f29d6e13098553e91f02193d51e40ecef93b80e2dbb1b9c14529ad: Status 404 returned error can't find the container with id 45c6216a75f29d6e13098553e91f02193d51e40ecef93b80e2dbb1b9c14529ad Nov 02 14:29:08.890185 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:08.890155 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" event={"ID":"3648aa77-bed0-4906-8f84-e11dceb7bf37","Type":"ContainerStarted","Data":"45c6216a75f29d6e13098553e91f02193d51e40ecef93b80e2dbb1b9c14529ad"} Nov 02 14:29:09.898081 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:09.898043 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" event={"ID":"3648aa77-bed0-4906-8f84-e11dceb7bf37","Type":"ContainerStarted","Data":"c08aeafd3c076da5fd56020567986ad1e55779c0fc44c079ee9284d8d2365221"} Nov 02 14:29:09.898557 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:09.898217 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" Nov 02 14:29:09.956969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:09.954085 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" podStartSLOduration=1.954039802 podStartE2EDuration="1.954039802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:29:09.942724849 +0000 UTC m=+1915.126572179" watchObservedRunningTime="2026-11-02 14:29:09.954039802 +0000 UTC m=+1915.137887139" Nov 02 14:29:15.840805 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:15.840774 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx" Nov 02 14:29:15.859758 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:15.859725 4866 scope.go:122] "RemoveContainer" containerID="23ab9581e067a2de8e8237c9cdf2328aebdf02af6dcca07a45b993b836d98ea3" Nov 02 14:29:18.970992 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:18.970934 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-8q5hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:29:18.979579 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:18.979535 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" podUID="1fbf701c-a1aa-4031-8112-4e69e00e103b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:29:20.908187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:20.908154 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" Nov 02 14:29:21.009183 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:21.009148 4866 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/e2e-cfl-622143-provider-cred: secret "e2e-cfl-622143-provider-cred" not found Nov 02 14:29:21.009183 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:21.009177 4866 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t: secret "e2e-cfl-622143-provider-cred" not found Nov 02 14:29:21.009428 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:21.009237 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-provider-credentials podName:3648aa77-bed0-4906-8f84-e11dceb7bf37 nodeName:}" failed. No retries permitted until 2026-11-02 14:29:21.509215923 +0000 UTC m=+1926.693063240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" (UID: "3648aa77-bed0-4906-8f84-e11dceb7bf37") : secret "e2e-cfl-622143-provider-cred" not found Nov 02 14:29:21.112228 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:21.112197 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6"] Nov 02 14:29:21.117410 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:21.117389 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" Nov 02 14:29:21.133157 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:21.133130 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"agc-cred-e06cd62782ae\"" Nov 02 14:29:21.135652 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:21.135625 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6"] Nov 02 14:29:21.211937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:21.211892 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5284z\" (UniqueName: \"kubernetes.io/projected/7785f817-f04f-49be-a16a-c54727168e70-kube-api-access-5284z\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6\" (UID: \"7785f817-f04f-49be-a16a-c54727168e70\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" Nov 02 14:29:21.212216 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:21.212048 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7785f817-f04f-49be-a16a-c54727168e70-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6\" (UID: \"7785f817-f04f-49be-a16a-c54727168e70\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" Nov 02 14:29:21.212310 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:21.212263 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7785f817-f04f-49be-a16a-c54727168e70-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6\" (UID: \"7785f817-f04f-49be-a16a-c54727168e70\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" Nov 02 14:29:21.212373 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:21.212316 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7785f817-f04f-49be-a16a-c54727168e70-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6\" (UID: \"7785f817-f04f-49be-a16a-c54727168e70\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" Nov 02 14:29:21.313848 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:21.313796 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7785f817-f04f-49be-a16a-c54727168e70-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6\" (UID: \"7785f817-f04f-49be-a16a-c54727168e70\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" Nov 02 14:29:21.314043 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:21.314011 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7785f817-f04f-49be-a16a-c54727168e70-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6\" (UID: \"7785f817-f04f-49be-a16a-c54727168e70\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" Nov 02 14:29:21.314091 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:21.314056 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7785f817-f04f-49be-a16a-c54727168e70-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6\" (UID: \"7785f817-f04f-49be-a16a-c54727168e70\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" Nov 02 14:29:21.314204 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:21.314185 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5284z\" (UniqueName: \"kubernetes.io/projected/7785f817-f04f-49be-a16a-c54727168e70-kube-api-access-5284z\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6\" (UID: \"7785f817-f04f-49be-a16a-c54727168e70\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" Nov 02 14:29:21.314273 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:21.314244 4866 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/agc-cred-e06cd62782ae: secret "agc-cred-e06cd62782ae" not found Nov 02 14:29:21.314327 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:21.314279 4866 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6: secret "agc-cred-e06cd62782ae" not found Nov 02 14:29:21.314368 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:21.314340 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7785f817-f04f-49be-a16a-c54727168e70-provider-credentials podName:7785f817-f04f-49be-a16a-c54727168e70 nodeName:}" failed. No retries permitted until 2026-11-02 14:29:21.814321091 +0000 UTC m=+1926.998168424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7785f817-f04f-49be-a16a-c54727168e70-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" (UID: "7785f817-f04f-49be-a16a-c54727168e70") : secret "agc-cred-e06cd62782ae" not found Nov 02 14:29:21.314680 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:21.314660 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7785f817-f04f-49be-a16a-c54727168e70-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6\" (UID: \"7785f817-f04f-49be-a16a-c54727168e70\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" Nov 02 14:29:21.314741 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:21.314723 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7785f817-f04f-49be-a16a-c54727168e70-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6\" (UID: \"7785f817-f04f-49be-a16a-c54727168e70\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" Nov 02 14:29:21.348524 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:21.348489 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5284z\" (UniqueName: \"kubernetes.io/projected/7785f817-f04f-49be-a16a-c54727168e70-kube-api-access-5284z\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6\" (UID: \"7785f817-f04f-49be-a16a-c54727168e70\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" Nov 02 14:29:21.517606 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:21.517522 4866 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/e2e-cfl-622143-provider-cred: secret "e2e-cfl-622143-provider-cred" not found Nov 02 14:29:21.517606 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:21.517551 4866 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t: secret "e2e-cfl-622143-provider-cred" not found Nov 02 14:29:21.517856 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:21.517616 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-provider-credentials podName:3648aa77-bed0-4906-8f84-e11dceb7bf37 nodeName:}" failed. No retries permitted until 2026-11-02 14:29:22.517595794 +0000 UTC m=+1927.701443110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" (UID: "3648aa77-bed0-4906-8f84-e11dceb7bf37") : secret "e2e-cfl-622143-provider-cred" not found Nov 02 14:29:21.821105 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:21.821003 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7785f817-f04f-49be-a16a-c54727168e70-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6\" (UID: \"7785f817-f04f-49be-a16a-c54727168e70\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" Nov 02 14:29:21.821291 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:21.821152 4866 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/agc-cred-e06cd62782ae: secret "agc-cred-e06cd62782ae" not found Nov 02 14:29:21.821291 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:21.821173 4866 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6: secret "agc-cred-e06cd62782ae" not found Nov 02 14:29:21.821291 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:21.821236 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7785f817-f04f-49be-a16a-c54727168e70-provider-credentials podName:7785f817-f04f-49be-a16a-c54727168e70 nodeName:}" failed. No retries permitted until 2026-11-02 14:29:22.821221068 +0000 UTC m=+1928.005068396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7785f817-f04f-49be-a16a-c54727168e70-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" (UID: "7785f817-f04f-49be-a16a-c54727168e70") : secret "agc-cred-e06cd62782ae" not found Nov 02 14:29:22.492220 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:22.492183 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6"] Nov 02 14:29:22.492639 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:22.492535 4866 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" podUID="7785f817-f04f-49be-a16a-c54727168e70" Nov 02 14:29:22.509504 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:22.509462 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t"] Nov 02 14:29:22.529860 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:22.529829 4866 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/e2e-cfl-622143-provider-cred: secret "e2e-cfl-622143-provider-cred" not found Nov 02 14:29:22.529860 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:22.529856 4866 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t: secret "e2e-cfl-622143-provider-cred" not found Nov 02 14:29:22.530130 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:22.529921 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-provider-credentials podName:3648aa77-bed0-4906-8f84-e11dceb7bf37 nodeName:}" failed. No retries permitted until 2026-11-02 14:29:24.529901201 +0000 UTC m=+1929.713748535 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" (UID: "3648aa77-bed0-4906-8f84-e11dceb7bf37") : secret "e2e-cfl-622143-provider-cred" not found Nov 02 14:29:22.834932 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:22.834771 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7785f817-f04f-49be-a16a-c54727168e70-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6\" (UID: \"7785f817-f04f-49be-a16a-c54727168e70\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" Nov 02 14:29:22.835234 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:22.834997 4866 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/agc-cred-e06cd62782ae: secret "agc-cred-e06cd62782ae" not found Nov 02 14:29:22.835234 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:22.835036 4866 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6: secret "agc-cred-e06cd62782ae" not found Nov 02 14:29:22.835234 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:22.835121 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7785f817-f04f-49be-a16a-c54727168e70-provider-credentials podName:7785f817-f04f-49be-a16a-c54727168e70 nodeName:}" failed. No retries permitted until 2026-11-02 14:29:24.835099344 +0000 UTC m=+1930.018946677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7785f817-f04f-49be-a16a-c54727168e70-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" (UID: "7785f817-f04f-49be-a16a-c54727168e70") : secret "agc-cred-e06cd62782ae" not found Nov 02 14:29:23.017223 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:23.017192 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" Nov 02 14:29:23.026673 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:23.026647 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" Nov 02 14:29:23.139410 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:23.139318 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7785f817-f04f-49be-a16a-c54727168e70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7785f817-f04f-49be-a16a-c54727168e70-plugins-config-volume\") pod \"7785f817-f04f-49be-a16a-c54727168e70\" (UID: \"7785f817-f04f-49be-a16a-c54727168e70\") " Nov 02 14:29:23.139410 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:23.139401 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7785f817-f04f-49be-a16a-c54727168e70-kube-api-access-5284z\" (UniqueName: \"kubernetes.io/projected/7785f817-f04f-49be-a16a-c54727168e70-kube-api-access-5284z\") pod \"7785f817-f04f-49be-a16a-c54727168e70\" (UID: \"7785f817-f04f-49be-a16a-c54727168e70\") " Nov 02 14:29:23.139633 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:23.139532 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7785f817-f04f-49be-a16a-c54727168e70-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7785f817-f04f-49be-a16a-c54727168e70-routing-overlay-volume\") pod \"7785f817-f04f-49be-a16a-c54727168e70\" (UID: \"7785f817-f04f-49be-a16a-c54727168e70\") " Nov 02 14:29:23.139899 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:23.139781 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7785f817-f04f-49be-a16a-c54727168e70-plugins-config-volume" pod "7785f817-f04f-49be-a16a-c54727168e70" (UID: "7785f817-f04f-49be-a16a-c54727168e70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:29:23.140057 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:23.139911 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7785f817-f04f-49be-a16a-c54727168e70-routing-overlay-volume" pod "7785f817-f04f-49be-a16a-c54727168e70" (UID: "7785f817-f04f-49be-a16a-c54727168e70"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:29:23.140121 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:23.140100 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7785f817-f04f-49be-a16a-c54727168e70-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:29:23.140161 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:23.140124 4866 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7785f817-f04f-49be-a16a-c54727168e70-routing-overlay-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:29:23.141739 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:23.141714 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7785f817-f04f-49be-a16a-c54727168e70-kube-api-access-5284z" pod "7785f817-f04f-49be-a16a-c54727168e70" (UID: "7785f817-f04f-49be-a16a-c54727168e70"). InnerVolumeSpecName "kube-api-access-5284z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:29:23.241256 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:23.241220 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5284z\" (UniqueName: \"kubernetes.io/projected/7785f817-f04f-49be-a16a-c54727168e70-kube-api-access-5284z\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:29:24.039100 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:24.039061 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6" Nov 02 14:29:24.119957 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:24.119909 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6"] Nov 02 14:29:24.124677 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:24.124647 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fnpcg6"] Nov 02 14:29:24.254440 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:24.254405 4866 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7785f817-f04f-49be-a16a-c54727168e70-provider-credentials\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:29:24.557745 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:24.557710 4866 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/e2e-cfl-622143-provider-cred: secret "e2e-cfl-622143-provider-cred" not found Nov 02 14:29:24.557745 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:24.557736 4866 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t: secret "e2e-cfl-622143-provider-cred" not found Nov 02 14:29:24.558055 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:24.557802 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-provider-credentials podName:3648aa77-bed0-4906-8f84-e11dceb7bf37 nodeName:}" failed. No retries permitted until 2026-11-02 14:29:28.55778439 +0000 UTC m=+1933.741631722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" (UID: "3648aa77-bed0-4906-8f84-e11dceb7bf37") : secret "e2e-cfl-622143-provider-cred" not found Nov 02 14:29:24.947023 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:24.946995 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7785f817-f04f-49be-a16a-c54727168e70" path="/var/lib/kubelet/pods/7785f817-f04f-49be-a16a-c54727168e70/volumes" Nov 02 14:29:27.510228 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:27.510132 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" podUID="3648aa77-bed0-4906-8f84-e11dceb7bf37" containerName="payload-processing" containerID="cri-o://c08aeafd3c076da5fd56020567986ad1e55779c0fc44c079ee9284d8d2365221" gracePeriod=25 Nov 02 14:29:27.785427 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:27.785400 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" Nov 02 14:29:27.797820 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:27.797793 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-kube-api-access-dhgmn\" (UniqueName: \"kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-kube-api-access-dhgmn\") pod \"3648aa77-bed0-4906-8f84-e11dceb7bf37\" (UID: \"3648aa77-bed0-4906-8f84-e11dceb7bf37\") " Nov 02 14:29:27.797998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:27.797882 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-provider-credentials\") pod \"3648aa77-bed0-4906-8f84-e11dceb7bf37\" (UID: \"3648aa77-bed0-4906-8f84-e11dceb7bf37\") " Nov 02 14:29:27.798087 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:27.798017 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3648aa77-bed0-4906-8f84-e11dceb7bf37-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3648aa77-bed0-4906-8f84-e11dceb7bf37-routing-overlay-volume\") pod \"3648aa77-bed0-4906-8f84-e11dceb7bf37\" (UID: \"3648aa77-bed0-4906-8f84-e11dceb7bf37\") " Nov 02 14:29:27.798087 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:27.798039 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3648aa77-bed0-4906-8f84-e11dceb7bf37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3648aa77-bed0-4906-8f84-e11dceb7bf37-plugins-config-volume\") pod \"3648aa77-bed0-4906-8f84-e11dceb7bf37\" (UID: \"3648aa77-bed0-4906-8f84-e11dceb7bf37\") " Nov 02 14:29:27.798437 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:27.798410 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3648aa77-bed0-4906-8f84-e11dceb7bf37-routing-overlay-volume" pod "3648aa77-bed0-4906-8f84-e11dceb7bf37" (UID: "3648aa77-bed0-4906-8f84-e11dceb7bf37"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:29:27.798554 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:27.798533 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3648aa77-bed0-4906-8f84-e11dceb7bf37-plugins-config-volume" pod "3648aa77-bed0-4906-8f84-e11dceb7bf37" (UID: "3648aa77-bed0-4906-8f84-e11dceb7bf37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:29:27.798628 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:27.798552 4866 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3648aa77-bed0-4906-8f84-e11dceb7bf37-routing-overlay-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:29:27.800700 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:27.800679 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-kube-api-access-dhgmn" pod "3648aa77-bed0-4906-8f84-e11dceb7bf37" (UID: "3648aa77-bed0-4906-8f84-e11dceb7bf37"). InnerVolumeSpecName "kube-api-access-dhgmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:29:27.800782 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:27.800709 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-provider-credentials" pod "3648aa77-bed0-4906-8f84-e11dceb7bf37" (UID: "3648aa77-bed0-4906-8f84-e11dceb7bf37"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:29:27.899596 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:27.899556 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhgmn\" (UniqueName: \"kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-kube-api-access-dhgmn\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:29:27.899596 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:27.899599 4866 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3648aa77-bed0-4906-8f84-e11dceb7bf37-provider-credentials\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:29:27.899840 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:27.899613 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3648aa77-bed0-4906-8f84-e11dceb7bf37-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:29:28.050992 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:28.050905 4866 generic.go:356] "Generic (PLEG): container finished" podID="3648aa77-bed0-4906-8f84-e11dceb7bf37" containerID="c08aeafd3c076da5fd56020567986ad1e55779c0fc44c079ee9284d8d2365221" exitCode=0 Nov 02 14:29:28.050992 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:28.050961 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" Nov 02 14:29:28.051192 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:28.050998 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" event={"ID":"3648aa77-bed0-4906-8f84-e11dceb7bf37","Type":"ContainerDied","Data":"c08aeafd3c076da5fd56020567986ad1e55779c0fc44c079ee9284d8d2365221"} Nov 02 14:29:28.051192 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:28.051058 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t" event={"ID":"3648aa77-bed0-4906-8f84-e11dceb7bf37","Type":"ContainerDied","Data":"45c6216a75f29d6e13098553e91f02193d51e40ecef93b80e2dbb1b9c14529ad"} Nov 02 14:29:28.051192 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:28.051089 4866 scope.go:122] "RemoveContainer" containerID="c08aeafd3c076da5fd56020567986ad1e55779c0fc44c079ee9284d8d2365221" Nov 02 14:29:28.061823 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:28.061800 4866 scope.go:122] "RemoveContainer" containerID="c08aeafd3c076da5fd56020567986ad1e55779c0fc44c079ee9284d8d2365221" Nov 02 14:29:28.062145 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:28.062124 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c08aeafd3c076da5fd56020567986ad1e55779c0fc44c079ee9284d8d2365221\": container with ID starting with c08aeafd3c076da5fd56020567986ad1e55779c0fc44c079ee9284d8d2365221 not found: ID does not exist" containerID="c08aeafd3c076da5fd56020567986ad1e55779c0fc44c079ee9284d8d2365221" Nov 02 14:29:28.062233 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:28.062151 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c08aeafd3c076da5fd56020567986ad1e55779c0fc44c079ee9284d8d2365221"} err="failed to get container status \"c08aeafd3c076da5fd56020567986ad1e55779c0fc44c079ee9284d8d2365221\": rpc error: code = NotFound desc = could not find container \"c08aeafd3c076da5fd56020567986ad1e55779c0fc44c079ee9284d8d2365221\": container with ID starting with c08aeafd3c076da5fd56020567986ad1e55779c0fc44c079ee9284d8d2365221 not found: ID does not exist" Nov 02 14:29:28.103253 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:28.103213 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t"] Nov 02 14:29:28.108038 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:28.108006 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcf924t"] Nov 02 14:29:28.948041 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:28.948011 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3648aa77-bed0-4906-8f84-e11dceb7bf37" path="/var/lib/kubelet/pods/3648aa77-bed0-4906-8f84-e11dceb7bf37/volumes" Nov 02 14:29:29.623743 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.623709 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk"] Nov 02 14:29:29.624303 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.624289 4866 cpu_manager.go:414] "containerMap: removing container" podUID="3648aa77-bed0-4906-8f84-e11dceb7bf37" containerName="payload-processing" Nov 02 14:29:29.624371 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.624304 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3648aa77-bed0-4906-8f84-e11dceb7bf37" containerName="payload-processing" Nov 02 14:29:29.624423 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.624416 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="3648aa77-bed0-4906-8f84-e11dceb7bf37" containerName="payload-processing" Nov 02 14:29:29.629355 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.629335 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" Nov 02 14:29:29.653229 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.653199 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d0c9b9-serving-cert\"" Nov 02 14:29:29.670975 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.670929 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk"] Nov 02 14:29:29.721961 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.721916 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cf97eaa-077b-4652-8907-acd8bfbadade-maas-api-tls\") pod \"maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk\" (UID: \"1cf97eaa-077b-4652-8907-acd8bfbadade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" Nov 02 14:29:29.721961 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.721965 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hss8\" (UniqueName: \"kubernetes.io/projected/1cf97eaa-077b-4652-8907-acd8bfbadade-kube-api-access-6hss8\") pod \"maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk\" (UID: \"1cf97eaa-077b-4652-8907-acd8bfbadade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" Nov 02 14:29:29.722217 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.722122 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf97eaa-077b-4652-8907-acd8bfbadade-openshift-service-ca\") pod \"maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk\" (UID: \"1cf97eaa-077b-4652-8907-acd8bfbadade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" Nov 02 14:29:29.722217 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.722160 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cf97eaa-077b-4652-8907-acd8bfbadade-metrics-certs\") pod \"maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk\" (UID: \"1cf97eaa-077b-4652-8907-acd8bfbadade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" Nov 02 14:29:29.823077 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.823036 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cf97eaa-077b-4652-8907-acd8bfbadade-maas-api-tls\") pod \"maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk\" (UID: \"1cf97eaa-077b-4652-8907-acd8bfbadade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" Nov 02 14:29:29.823077 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.823083 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hss8\" (UniqueName: \"kubernetes.io/projected/1cf97eaa-077b-4652-8907-acd8bfbadade-kube-api-access-6hss8\") pod \"maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk\" (UID: \"1cf97eaa-077b-4652-8907-acd8bfbadade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" Nov 02 14:29:29.823339 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.823188 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf97eaa-077b-4652-8907-acd8bfbadade-openshift-service-ca\") pod \"maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk\" (UID: \"1cf97eaa-077b-4652-8907-acd8bfbadade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" Nov 02 14:29:29.823339 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.823214 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cf97eaa-077b-4652-8907-acd8bfbadade-metrics-certs\") pod \"maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk\" (UID: \"1cf97eaa-077b-4652-8907-acd8bfbadade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" Nov 02 14:29:29.823936 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.823913 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf97eaa-077b-4652-8907-acd8bfbadade-openshift-service-ca\") pod \"maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk\" (UID: \"1cf97eaa-077b-4652-8907-acd8bfbadade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" Nov 02 14:29:29.825547 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.825526 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cf97eaa-077b-4652-8907-acd8bfbadade-maas-api-tls\") pod \"maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk\" (UID: \"1cf97eaa-077b-4652-8907-acd8bfbadade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" Nov 02 14:29:29.825653 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.825607 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cf97eaa-077b-4652-8907-acd8bfbadade-metrics-certs\") pod \"maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk\" (UID: \"1cf97eaa-077b-4652-8907-acd8bfbadade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" Nov 02 14:29:29.853163 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.853115 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hss8\" (UniqueName: \"kubernetes.io/projected/1cf97eaa-077b-4652-8907-acd8bfbadade-kube-api-access-6hss8\") pod \"maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk\" (UID: \"1cf97eaa-077b-4652-8907-acd8bfbadade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" Nov 02 14:29:29.942680 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:29.942648 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" Nov 02 14:29:30.254585 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:29:30.254557 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf97eaa_077b_4652_8907_acd8bfbadade.slice/crio-5afa1babf7de10f73e93cc7c156735e90d72292752f8a83a16fa93dbcc456f40 WatchSource:0}: Error finding container 5afa1babf7de10f73e93cc7c156735e90d72292752f8a83a16fa93dbcc456f40: Status 404 returned error can't find the container with id 5afa1babf7de10f73e93cc7c156735e90d72292752f8a83a16fa93dbcc456f40 Nov 02 14:29:30.254937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:30.254916 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk"] Nov 02 14:29:31.072187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:31.072097 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" event={"ID":"1cf97eaa-077b-4652-8907-acd8bfbadade","Type":"ContainerStarted","Data":"21178d4aff6b0e12402bfee2d69f9bc464867eb4928f5ac617b0c2ec7d17b6b3"} Nov 02 14:29:31.072187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:31.072156 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" event={"ID":"1cf97eaa-077b-4652-8907-acd8bfbadade","Type":"ContainerStarted","Data":"5afa1babf7de10f73e93cc7c156735e90d72292752f8a83a16fa93dbcc456f40"} Nov 02 14:29:31.072187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:31.072176 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" Nov 02 14:29:31.117844 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:31.117789 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" podStartSLOduration=1.59380101 podStartE2EDuration="2.117774039s" totalImagesPullingTime="523.973029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:29:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:29:31.114177614 +0000 UTC m=+1936.298024950" watchObservedRunningTime="2026-11-02 14:29:31.117774039 +0000 UTC m=+1936.301621375" Nov 02 14:29:36.009754 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:29:36.009713 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:29:37.084694 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:37.084665 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" Nov 02 14:29:58.929732 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:58.929693 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr"] Nov 02 14:29:58.930199 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:29:58.929976 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" podUID="bc607e50-88ca-40d8-9a3b-1b1335f74098" containerName="istio-proxy" containerID="cri-o://1ada60c8c2ff70b00a4ecb28c709dbc745c8b8e4fe562b55d3020b1f90dad329" gracePeriod=30 Nov 02 14:30:00.302325 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.302269 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b"] Nov 02 14:30:00.307740 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.307714 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8"] Nov 02 14:30:00.307911 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.307822 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" Nov 02 14:30:00.312260 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.312232 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" Nov 02 14:30:00.334416 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.334389 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:30:00.340725 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.340694 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk"] Nov 02 14:30:00.346000 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.345930 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" Nov 02 14:30:00.348878 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.348831 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b"] Nov 02 14:30:00.354718 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.354687 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8"] Nov 02 14:30:00.389841 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.389809 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk"] Nov 02 14:30:00.416595 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.416556 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de02a12c-740c-45e4-bf68-ec8151644875-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b\" (UID: \"de02a12c-740c-45e4-bf68-ec8151644875\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" Nov 02 14:30:00.416770 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.416676 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7srm4\" (UniqueName: \"kubernetes.io/projected/3567c919-48d4-4f83-9a1b-32031ddc297f-kube-api-access-7srm4\") pod \"maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8\" (UID: \"3567c919-48d4-4f83-9a1b-32031ddc297f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" Nov 02 14:30:00.416825 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.416808 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x64r\" (UniqueName: \"kubernetes.io/projected/de02a12c-740c-45e4-bf68-ec8151644875-kube-api-access-8x64r\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b\" (UID: \"de02a12c-740c-45e4-bf68-ec8151644875\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" Nov 02 14:30:00.416870 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.416835 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3567c919-48d4-4f83-9a1b-32031ddc297f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8\" (UID: \"3567c919-48d4-4f83-9a1b-32031ddc297f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" Nov 02 14:30:00.518075 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.518041 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e672770-0503-4abb-8716-73be17d23f2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk\" (UID: \"5e672770-0503-4abb-8716-73be17d23f2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" Nov 02 14:30:00.518257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.518090 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x64r\" (UniqueName: \"kubernetes.io/projected/de02a12c-740c-45e4-bf68-ec8151644875-kube-api-access-8x64r\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b\" (UID: \"de02a12c-740c-45e4-bf68-ec8151644875\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" Nov 02 14:30:00.518257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.518112 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3567c919-48d4-4f83-9a1b-32031ddc297f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8\" (UID: \"3567c919-48d4-4f83-9a1b-32031ddc297f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" Nov 02 14:30:00.518257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.518224 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bhv9\" (UniqueName: \"kubernetes.io/projected/5e672770-0503-4abb-8716-73be17d23f2c-kube-api-access-5bhv9\") pod \"maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk\" (UID: \"5e672770-0503-4abb-8716-73be17d23f2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" Nov 02 14:30:00.518428 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.518282 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de02a12c-740c-45e4-bf68-ec8151644875-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b\" (UID: \"de02a12c-740c-45e4-bf68-ec8151644875\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" Nov 02 14:30:00.518550 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.518430 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7srm4\" (UniqueName: \"kubernetes.io/projected/3567c919-48d4-4f83-9a1b-32031ddc297f-kube-api-access-7srm4\") pod \"maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8\" (UID: \"3567c919-48d4-4f83-9a1b-32031ddc297f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" Nov 02 14:30:00.518822 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.518804 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3567c919-48d4-4f83-9a1b-32031ddc297f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8\" (UID: \"3567c919-48d4-4f83-9a1b-32031ddc297f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" Nov 02 14:30:00.518894 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.518876 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de02a12c-740c-45e4-bf68-ec8151644875-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b\" (UID: \"de02a12c-740c-45e4-bf68-ec8151644875\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" Nov 02 14:30:00.555842 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.555763 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7srm4\" (UniqueName: \"kubernetes.io/projected/3567c919-48d4-4f83-9a1b-32031ddc297f-kube-api-access-7srm4\") pod \"maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8\" (UID: \"3567c919-48d4-4f83-9a1b-32031ddc297f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" Nov 02 14:30:00.555842 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.555777 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x64r\" (UniqueName: \"kubernetes.io/projected/de02a12c-740c-45e4-bf68-ec8151644875-kube-api-access-8x64r\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b\" (UID: \"de02a12c-740c-45e4-bf68-ec8151644875\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" Nov 02 14:30:00.619839 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.619803 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e672770-0503-4abb-8716-73be17d23f2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk\" (UID: \"5e672770-0503-4abb-8716-73be17d23f2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" Nov 02 14:30:00.620027 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.619922 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bhv9\" (UniqueName: \"kubernetes.io/projected/5e672770-0503-4abb-8716-73be17d23f2c-kube-api-access-5bhv9\") pod \"maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk\" (UID: \"5e672770-0503-4abb-8716-73be17d23f2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" Nov 02 14:30:00.620529 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.620509 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e672770-0503-4abb-8716-73be17d23f2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk\" (UID: \"5e672770-0503-4abb-8716-73be17d23f2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" Nov 02 14:30:00.627509 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.627474 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" Nov 02 14:30:00.635595 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.635567 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" Nov 02 14:30:00.644130 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.644099 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bhv9\" (UniqueName: \"kubernetes.io/projected/5e672770-0503-4abb-8716-73be17d23f2c-kube-api-access-5bhv9\") pod \"maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk\" (UID: \"5e672770-0503-4abb-8716-73be17d23f2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" Nov 02 14:30:00.689131 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.687351 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" Nov 02 14:30:00.944776 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:30:00.944742 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde02a12c_740c_45e4_bf68_ec8151644875.slice/crio-552c96a192a0eebca95ee6e4c6e9b7bf88d64014d86edfaf3fa7c1e3f13cb525 WatchSource:0}: Error finding container 552c96a192a0eebca95ee6e4c6e9b7bf88d64014d86edfaf3fa7c1e3f13cb525: Status 404 returned error can't find the container with id 552c96a192a0eebca95ee6e4c6e9b7bf88d64014d86edfaf3fa7c1e3f13cb525 Nov 02 14:30:00.949999 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.949975 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b"] Nov 02 14:30:00.950776 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:00.950753 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8"] Nov 02 14:30:00.956705 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:30:00.956682 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3567c919_48d4_4f83_9a1b_32031ddc297f.slice/crio-4b7b792650c143ecdfd5c92643aeeba9cdfb2e0442c80ac4e114fafda73f1063 WatchSource:0}: Error finding container 4b7b792650c143ecdfd5c92643aeeba9cdfb2e0442c80ac4e114fafda73f1063: Status 404 returned error can't find the container with id 4b7b792650c143ecdfd5c92643aeeba9cdfb2e0442c80ac4e114fafda73f1063 Nov 02 14:30:01.029485 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:01.029455 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk"] Nov 02 14:30:01.100730 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:30:01.100695 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e672770_0503_4abb_8716_73be17d23f2c.slice/crio-76eaf76abb8d178137102a30ad0bdfd5abce40d502c3043e812532f0102ba63b WatchSource:0}: Error finding container 76eaf76abb8d178137102a30ad0bdfd5abce40d502c3043e812532f0102ba63b: Status 404 returned error can't find the container with id 76eaf76abb8d178137102a30ad0bdfd5abce40d502c3043e812532f0102ba63b Nov 02 14:30:01.282308 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:01.282264 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" event={"ID":"3567c919-48d4-4f83-9a1b-32031ddc297f","Type":"ContainerStarted","Data":"4b7b792650c143ecdfd5c92643aeeba9cdfb2e0442c80ac4e114fafda73f1063"} Nov 02 14:30:01.283711 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:01.283676 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" event={"ID":"5e672770-0503-4abb-8716-73be17d23f2c","Type":"ContainerStarted","Data":"76eaf76abb8d178137102a30ad0bdfd5abce40d502c3043e812532f0102ba63b"} Nov 02 14:30:01.284984 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:01.284937 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" event={"ID":"de02a12c-740c-45e4-bf68-ec8151644875","Type":"ContainerStarted","Data":"552c96a192a0eebca95ee6e4c6e9b7bf88d64014d86edfaf3fa7c1e3f13cb525"} Nov 02 14:30:02.293669 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:02.293633 4866 generic.go:356] "Generic (PLEG): container finished" podID="3567c919-48d4-4f83-9a1b-32031ddc297f" containerID="bc8e442e73f9d5e2a3117821f74a9886033f008c0463d3dbf2ebfb7fa0129085" exitCode=60 Nov 02 14:30:02.294168 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:02.293725 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" event={"ID":"3567c919-48d4-4f83-9a1b-32031ddc297f","Type":"ContainerDied","Data":"bc8e442e73f9d5e2a3117821f74a9886033f008c0463d3dbf2ebfb7fa0129085"} Nov 02 14:30:02.294168 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:02.294100 4866 scope.go:122] "RemoveContainer" containerID="bc8e442e73f9d5e2a3117821f74a9886033f008c0463d3dbf2ebfb7fa0129085" Nov 02 14:30:02.295808 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:02.295783 4866 generic.go:356] "Generic (PLEG): container finished" podID="5e672770-0503-4abb-8716-73be17d23f2c" containerID="b6fc372afb17979244b17fbe4201e2f312937fe5bcee71e97e53ff0c914d3547" exitCode=60 Nov 02 14:30:02.295926 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:02.295836 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" event={"ID":"5e672770-0503-4abb-8716-73be17d23f2c","Type":"ContainerDied","Data":"b6fc372afb17979244b17fbe4201e2f312937fe5bcee71e97e53ff0c914d3547"} Nov 02 14:30:02.296409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:02.296214 4866 scope.go:122] "RemoveContainer" containerID="b6fc372afb17979244b17fbe4201e2f312937fe5bcee71e97e53ff0c914d3547" Nov 02 14:30:02.297998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:02.297717 4866 generic.go:356] "Generic (PLEG): container finished" podID="de02a12c-740c-45e4-bf68-ec8151644875" containerID="d97f6cf43d4bb7ca56ddd63a40b1a19e547183e1493bf85d3dad0741955152ee" exitCode=60 Nov 02 14:30:02.297998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:02.297840 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" event={"ID":"de02a12c-740c-45e4-bf68-ec8151644875","Type":"ContainerDied","Data":"d97f6cf43d4bb7ca56ddd63a40b1a19e547183e1493bf85d3dad0741955152ee"} Nov 02 14:30:02.298191 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:02.298078 4866 scope.go:122] "RemoveContainer" containerID="d97f6cf43d4bb7ca56ddd63a40b1a19e547183e1493bf85d3dad0741955152ee" Nov 02 14:30:03.309567 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:03.309539 4866 generic.go:356] "Generic (PLEG): container finished" podID="3567c919-48d4-4f83-9a1b-32031ddc297f" containerID="a4d24696027bfb0c7f81e42a764e3be7650a5cd23f02db0b83b80e95140a2705" exitCode=60 Nov 02 14:30:03.310017 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:03.309621 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" event={"ID":"3567c919-48d4-4f83-9a1b-32031ddc297f","Type":"ContainerDied","Data":"a4d24696027bfb0c7f81e42a764e3be7650a5cd23f02db0b83b80e95140a2705"} Nov 02 14:30:03.310017 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:03.309694 4866 scope.go:122] "RemoveContainer" containerID="bc8e442e73f9d5e2a3117821f74a9886033f008c0463d3dbf2ebfb7fa0129085" Nov 02 14:30:03.310017 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:03.309933 4866 scope.go:122] "RemoveContainer" containerID="a4d24696027bfb0c7f81e42a764e3be7650a5cd23f02db0b83b80e95140a2705" Nov 02 14:30:03.310325 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:03.310300 4866 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8_odh-ai-gateway-infra(3567c919-48d4-4f83-9a1b-32031ddc297f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" podUID="3567c919-48d4-4f83-9a1b-32031ddc297f" Nov 02 14:30:03.311790 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:03.311769 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" event={"ID":"5e672770-0503-4abb-8716-73be17d23f2c","Type":"ContainerStarted","Data":"a3fd2b2edffc42950cf061642aa67366306aea711b663f923b91e219f5fe2866"} Nov 02 14:30:03.312068 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:03.312050 4866 scope.go:122] "RemoveContainer" containerID="a3fd2b2edffc42950cf061642aa67366306aea711b663f923b91e219f5fe2866" Nov 02 14:30:03.312398 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:03.312329 4866 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk_odh-ai-gateway-infra(5e672770-0503-4abb-8716-73be17d23f2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" podUID="5e672770-0503-4abb-8716-73be17d23f2c" Nov 02 14:30:03.313919 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:03.313899 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" event={"ID":"de02a12c-740c-45e4-bf68-ec8151644875","Type":"ContainerStarted","Data":"d70003908fff2691596aeb413b69ab078a734893efba62ddcb30f8ebde22960d"} Nov 02 14:30:03.394444 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:03.394387 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" podStartSLOduration=3.394374004 podStartE2EDuration="3.394374004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:03.393610984 +0000 UTC m=+1968.577458321" watchObservedRunningTime="2026-11-02 14:30:03.394374004 +0000 UTC m=+1968.578221340" Nov 02 14:30:04.217721 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.217683 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:30:04.321134 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.321102 4866 generic.go:356] "Generic (PLEG): container finished" podID="bc607e50-88ca-40d8-9a3b-1b1335f74098" containerID="1ada60c8c2ff70b00a4ecb28c709dbc745c8b8e4fe562b55d3020b1f90dad329" exitCode=0 Nov 02 14:30:04.321652 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.321151 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" Nov 02 14:30:04.321652 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.321191 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" event={"ID":"bc607e50-88ca-40d8-9a3b-1b1335f74098","Type":"ContainerDied","Data":"1ada60c8c2ff70b00a4ecb28c709dbc745c8b8e4fe562b55d3020b1f90dad329"} Nov 02 14:30:04.321652 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.321245 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr" event={"ID":"bc607e50-88ca-40d8-9a3b-1b1335f74098","Type":"ContainerDied","Data":"b3078b404cf050b9a10e3e925011cb9456701487667c5c1fc22c2d7b0c8d7fec"} Nov 02 14:30:04.321652 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.321275 4866 scope.go:122] "RemoveContainer" containerID="1ada60c8c2ff70b00a4ecb28c709dbc745c8b8e4fe562b55d3020b1f90dad329" Nov 02 14:30:04.323577 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.323548 4866 scope.go:122] "RemoveContainer" containerID="a4d24696027bfb0c7f81e42a764e3be7650a5cd23f02db0b83b80e95140a2705" Nov 02 14:30:04.323890 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:04.323865 4866 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8_odh-ai-gateway-infra(3567c919-48d4-4f83-9a1b-32031ddc297f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" podUID="3567c919-48d4-4f83-9a1b-32031ddc297f" Nov 02 14:30:04.325606 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.325582 4866 generic.go:356] "Generic (PLEG): container finished" podID="5e672770-0503-4abb-8716-73be17d23f2c" containerID="a3fd2b2edffc42950cf061642aa67366306aea711b663f923b91e219f5fe2866" exitCode=60 Nov 02 14:30:04.326127 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.325874 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" event={"ID":"5e672770-0503-4abb-8716-73be17d23f2c","Type":"ContainerDied","Data":"a3fd2b2edffc42950cf061642aa67366306aea711b663f923b91e219f5fe2866"} Nov 02 14:30:04.326127 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.326044 4866 scope.go:122] "RemoveContainer" containerID="a3fd2b2edffc42950cf061642aa67366306aea711b663f923b91e219f5fe2866" Nov 02 14:30:04.326376 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:04.326349 4866 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk_odh-ai-gateway-infra(5e672770-0503-4abb-8716-73be17d23f2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" podUID="5e672770-0503-4abb-8716-73be17d23f2c" Nov 02 14:30:04.328050 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.328024 4866 generic.go:356] "Generic (PLEG): container finished" podID="de02a12c-740c-45e4-bf68-ec8151644875" containerID="d70003908fff2691596aeb413b69ab078a734893efba62ddcb30f8ebde22960d" exitCode=60 Nov 02 14:30:04.328187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.328171 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" event={"ID":"de02a12c-740c-45e4-bf68-ec8151644875","Type":"ContainerDied","Data":"d70003908fff2691596aeb413b69ab078a734893efba62ddcb30f8ebde22960d"} Nov 02 14:30:04.328326 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.328311 4866 scope.go:122] "RemoveContainer" containerID="d70003908fff2691596aeb413b69ab078a734893efba62ddcb30f8ebde22960d" Nov 02 14:30:04.328552 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:04.328534 4866 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b_odh-ai-gateway-infra(de02a12c-740c-45e4-bf68-ec8151644875)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" podUID="de02a12c-740c-45e4-bf68-ec8151644875" Nov 02 14:30:04.332584 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.332565 4866 scope.go:122] "RemoveContainer" containerID="1ada60c8c2ff70b00a4ecb28c709dbc745c8b8e4fe562b55d3020b1f90dad329" Nov 02 14:30:04.332875 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:04.332853 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ada60c8c2ff70b00a4ecb28c709dbc745c8b8e4fe562b55d3020b1f90dad329\": container with ID starting with 1ada60c8c2ff70b00a4ecb28c709dbc745c8b8e4fe562b55d3020b1f90dad329 not found: ID does not exist" containerID="1ada60c8c2ff70b00a4ecb28c709dbc745c8b8e4fe562b55d3020b1f90dad329" Nov 02 14:30:04.333006 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.332881 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ada60c8c2ff70b00a4ecb28c709dbc745c8b8e4fe562b55d3020b1f90dad329"} err="failed to get container status \"1ada60c8c2ff70b00a4ecb28c709dbc745c8b8e4fe562b55d3020b1f90dad329\": rpc error: code = NotFound desc = could not find container \"1ada60c8c2ff70b00a4ecb28c709dbc745c8b8e4fe562b55d3020b1f90dad329\": container with ID starting with 1ada60c8c2ff70b00a4ecb28c709dbc745c8b8e4fe562b55d3020b1f90dad329 not found: ID does not exist" Nov 02 14:30:04.333006 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.332897 4866 scope.go:122] "RemoveContainer" containerID="b6fc372afb17979244b17fbe4201e2f312937fe5bcee71e97e53ff0c914d3547" Nov 02 14:30:04.346051 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.346023 4866 scope.go:122] "RemoveContainer" containerID="d97f6cf43d4bb7ca56ddd63a40b1a19e547183e1493bf85d3dad0741955152ee" Nov 02 14:30:04.366268 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.366216 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc607e50-88ca-40d8-9a3b-1b1335f74098-kube-api-access-cb7q6\" (UniqueName: \"kubernetes.io/projected/bc607e50-88ca-40d8-9a3b-1b1335f74098-kube-api-access-cb7q6\") pod \"bc607e50-88ca-40d8-9a3b-1b1335f74098\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " Nov 02 14:30:04.366397 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.366303 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-credential-socket\") pod \"bc607e50-88ca-40d8-9a3b-1b1335f74098\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " Nov 02 14:30:04.366397 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.366335 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-workload-certs\") pod \"bc607e50-88ca-40d8-9a3b-1b1335f74098\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " Nov 02 14:30:04.366499 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.366396 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-workload-socket\") pod \"bc607e50-88ca-40d8-9a3b-1b1335f74098\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " Nov 02 14:30:04.366499 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.366416 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-data\") pod \"bc607e50-88ca-40d8-9a3b-1b1335f74098\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " Nov 02 14:30:04.366499 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.366468 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc607e50-88ca-40d8-9a3b-1b1335f74098-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc607e50-88ca-40d8-9a3b-1b1335f74098-istiod-ca-cert\") pod \"bc607e50-88ca-40d8-9a3b-1b1335f74098\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " Nov 02 14:30:04.366656 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.366497 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-podinfo\") pod \"bc607e50-88ca-40d8-9a3b-1b1335f74098\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " Nov 02 14:30:04.366656 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.366581 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-credential-socket" pod "bc607e50-88ca-40d8-9a3b-1b1335f74098" (UID: "bc607e50-88ca-40d8-9a3b-1b1335f74098"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:30:04.366656 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.366627 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-envoy\") pod \"bc607e50-88ca-40d8-9a3b-1b1335f74098\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " Nov 02 14:30:04.366790 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.366673 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-token\" (UniqueName: \"kubernetes.io/projected/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-token\") pod \"bc607e50-88ca-40d8-9a3b-1b1335f74098\" (UID: \"bc607e50-88ca-40d8-9a3b-1b1335f74098\") " Nov 02 14:30:04.366790 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.366681 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-workload-certs" pod "bc607e50-88ca-40d8-9a3b-1b1335f74098" (UID: "bc607e50-88ca-40d8-9a3b-1b1335f74098"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:30:04.366790 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.366708 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-workload-socket" pod "bc607e50-88ca-40d8-9a3b-1b1335f74098" (UID: "bc607e50-88ca-40d8-9a3b-1b1335f74098"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:30:04.366790 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.366727 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-data" pod "bc607e50-88ca-40d8-9a3b-1b1335f74098" (UID: "bc607e50-88ca-40d8-9a3b-1b1335f74098"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:30:04.367198 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.367170 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:04.367320 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.367199 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:04.367320 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.367215 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:04.367320 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.367213 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc607e50-88ca-40d8-9a3b-1b1335f74098-istiod-ca-cert" pod "bc607e50-88ca-40d8-9a3b-1b1335f74098" (UID: "bc607e50-88ca-40d8-9a3b-1b1335f74098"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:04.367320 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.367229 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:04.368605 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.368583 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc607e50-88ca-40d8-9a3b-1b1335f74098-kube-api-access-cb7q6" pod "bc607e50-88ca-40d8-9a3b-1b1335f74098" (UID: "bc607e50-88ca-40d8-9a3b-1b1335f74098"). InnerVolumeSpecName "kube-api-access-cb7q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:04.368712 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.368623 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-podinfo" pod "bc607e50-88ca-40d8-9a3b-1b1335f74098" (UID: "bc607e50-88ca-40d8-9a3b-1b1335f74098"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:30:04.369092 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.369070 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-token" pod "bc607e50-88ca-40d8-9a3b-1b1335f74098" (UID: "bc607e50-88ca-40d8-9a3b-1b1335f74098"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:04.369163 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.369070 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-envoy" pod "bc607e50-88ca-40d8-9a3b-1b1335f74098" (UID: "bc607e50-88ca-40d8-9a3b-1b1335f74098"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:30:04.468766 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.468677 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:04.468766 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.468710 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:04.468766 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.468725 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb7q6\" (UniqueName: \"kubernetes.io/projected/bc607e50-88ca-40d8-9a3b-1b1335f74098-kube-api-access-cb7q6\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:04.468766 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.468740 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc607e50-88ca-40d8-9a3b-1b1335f74098-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:04.468766 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.468753 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc607e50-88ca-40d8-9a3b-1b1335f74098-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:04.669124 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.669085 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr"] Nov 02 14:30:04.677667 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.677634 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-tf5fr"] Nov 02 14:30:04.947912 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:04.947881 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc607e50-88ca-40d8-9a3b-1b1335f74098" path="/var/lib/kubelet/pods/bc607e50-88ca-40d8-9a3b-1b1335f74098/volumes" Nov 02 14:30:05.338348 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:05.338250 4866 scope.go:122] "RemoveContainer" containerID="a3fd2b2edffc42950cf061642aa67366306aea711b663f923b91e219f5fe2866" Nov 02 14:30:05.338828 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:05.338600 4866 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk_odh-ai-gateway-infra(5e672770-0503-4abb-8716-73be17d23f2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" podUID="5e672770-0503-4abb-8716-73be17d23f2c" Nov 02 14:30:05.339789 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:05.339771 4866 scope.go:122] "RemoveContainer" containerID="d70003908fff2691596aeb413b69ab078a734893efba62ddcb30f8ebde22960d" Nov 02 14:30:05.340043 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:05.340023 4866 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b_odh-ai-gateway-infra(de02a12c-740c-45e4-bf68-ec8151644875)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" podUID="de02a12c-740c-45e4-bf68-ec8151644875" Nov 02 14:30:06.011002 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:06.010969 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:30:07.904799 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:07.904768 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf"] Nov 02 14:30:07.946342 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:07.946309 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl"] Nov 02 14:30:12.905885 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:12.905786 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf" podUID="c2e5e98d-359e-4417-a474-c3678ab12022" containerName="payload-processing" containerID="cri-o://8ccd098fcf67d43285995c2e9410a210e8509636f32b2af47b3dcb030792469d" gracePeriod=25 Nov 02 14:30:12.946646 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:12.946578 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl" podUID="427c3914-9b8b-467f-bcb6-ccb99ee098cf" containerName="payload-pre-processing" containerID="cri-o://87d9c913b7e1fa0faa47f0a777953f201c0c4e429487fa15003149426ea37937" gracePeriod=25 Nov 02 14:30:13.214708 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.214680 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf" Nov 02 14:30:13.242308 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.242276 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl" Nov 02 14:30:13.284388 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.284351 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/427c3914-9b8b-467f-bcb6-ccb99ee098cf-kube-api-access-cjr5s\" (UniqueName: \"kubernetes.io/projected/427c3914-9b8b-467f-bcb6-ccb99ee098cf-kube-api-access-cjr5s\") pod \"427c3914-9b8b-467f-bcb6-ccb99ee098cf\" (UID: \"427c3914-9b8b-467f-bcb6-ccb99ee098cf\") " Nov 02 14:30:13.284663 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.284494 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/427c3914-9b8b-467f-bcb6-ccb99ee098cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427c3914-9b8b-467f-bcb6-ccb99ee098cf-plugins-config-volume\") pod \"427c3914-9b8b-467f-bcb6-ccb99ee098cf\" (UID: \"427c3914-9b8b-467f-bcb6-ccb99ee098cf\") " Nov 02 14:30:13.284663 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.284654 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2e5e98d-359e-4417-a474-c3678ab12022-kube-api-access-vxpls\" (UniqueName: \"kubernetes.io/projected/c2e5e98d-359e-4417-a474-c3678ab12022-kube-api-access-vxpls\") pod \"c2e5e98d-359e-4417-a474-c3678ab12022\" (UID: \"c2e5e98d-359e-4417-a474-c3678ab12022\") " Nov 02 14:30:13.284784 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.284718 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2e5e98d-359e-4417-a474-c3678ab12022-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e5e98d-359e-4417-a474-c3678ab12022-plugins-config-volume\") pod \"c2e5e98d-359e-4417-a474-c3678ab12022\" (UID: \"c2e5e98d-359e-4417-a474-c3678ab12022\") " Nov 02 14:30:13.285036 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.285008 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/427c3914-9b8b-467f-bcb6-ccb99ee098cf-plugins-config-volume" pod "427c3914-9b8b-467f-bcb6-ccb99ee098cf" (UID: "427c3914-9b8b-467f-bcb6-ccb99ee098cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:13.285307 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.285265 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427c3914-9b8b-467f-bcb6-ccb99ee098cf-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:13.285476 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.285438 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e5e98d-359e-4417-a474-c3678ab12022-plugins-config-volume" pod "c2e5e98d-359e-4417-a474-c3678ab12022" (UID: "c2e5e98d-359e-4417-a474-c3678ab12022"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:13.286765 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.286743 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427c3914-9b8b-467f-bcb6-ccb99ee098cf-kube-api-access-cjr5s" pod "427c3914-9b8b-467f-bcb6-ccb99ee098cf" (UID: "427c3914-9b8b-467f-bcb6-ccb99ee098cf"). InnerVolumeSpecName "kube-api-access-cjr5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:13.287168 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.287151 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e5e98d-359e-4417-a474-c3678ab12022-kube-api-access-vxpls" pod "c2e5e98d-359e-4417-a474-c3678ab12022" (UID: "c2e5e98d-359e-4417-a474-c3678ab12022"). InnerVolumeSpecName "kube-api-access-vxpls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:13.389237 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.389199 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjr5s\" (UniqueName: \"kubernetes.io/projected/427c3914-9b8b-467f-bcb6-ccb99ee098cf-kube-api-access-cjr5s\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:13.389237 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.389236 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxpls\" (UniqueName: \"kubernetes.io/projected/c2e5e98d-359e-4417-a474-c3678ab12022-kube-api-access-vxpls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:13.389500 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.389254 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e5e98d-359e-4417-a474-c3678ab12022-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:13.398940 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.398906 4866 generic.go:356] "Generic (PLEG): container finished" podID="c2e5e98d-359e-4417-a474-c3678ab12022" containerID="8ccd098fcf67d43285995c2e9410a210e8509636f32b2af47b3dcb030792469d" exitCode=0 Nov 02 14:30:13.399133 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.398984 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf" Nov 02 14:30:13.399133 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.399013 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf" event={"ID":"c2e5e98d-359e-4417-a474-c3678ab12022","Type":"ContainerDied","Data":"8ccd098fcf67d43285995c2e9410a210e8509636f32b2af47b3dcb030792469d"} Nov 02 14:30:13.399133 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.399114 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf" event={"ID":"c2e5e98d-359e-4417-a474-c3678ab12022","Type":"ContainerDied","Data":"c907a5638053dd9f6905b5d3f840e88a0af8f0396087cfca7c47556d20f7ab4d"} Nov 02 14:30:13.399254 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.399147 4866 scope.go:122] "RemoveContainer" containerID="8ccd098fcf67d43285995c2e9410a210e8509636f32b2af47b3dcb030792469d" Nov 02 14:30:13.400914 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.400887 4866 generic.go:356] "Generic (PLEG): container finished" podID="427c3914-9b8b-467f-bcb6-ccb99ee098cf" containerID="87d9c913b7e1fa0faa47f0a777953f201c0c4e429487fa15003149426ea37937" exitCode=0 Nov 02 14:30:13.401041 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.400927 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl" event={"ID":"427c3914-9b8b-467f-bcb6-ccb99ee098cf","Type":"ContainerDied","Data":"87d9c913b7e1fa0faa47f0a777953f201c0c4e429487fa15003149426ea37937"} Nov 02 14:30:13.401041 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.400986 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl" event={"ID":"427c3914-9b8b-467f-bcb6-ccb99ee098cf","Type":"ContainerDied","Data":"ea9b893849036b211d240775d8b0da277ed3a369f0988302ba713a29fd192e60"} Nov 02 14:30:13.401124 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.401095 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl" Nov 02 14:30:13.409591 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.409574 4866 scope.go:122] "RemoveContainer" containerID="8ccd098fcf67d43285995c2e9410a210e8509636f32b2af47b3dcb030792469d" Nov 02 14:30:13.409874 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:13.409853 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ccd098fcf67d43285995c2e9410a210e8509636f32b2af47b3dcb030792469d\": container with ID starting with 8ccd098fcf67d43285995c2e9410a210e8509636f32b2af47b3dcb030792469d not found: ID does not exist" containerID="8ccd098fcf67d43285995c2e9410a210e8509636f32b2af47b3dcb030792469d" Nov 02 14:30:13.409977 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.409880 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ccd098fcf67d43285995c2e9410a210e8509636f32b2af47b3dcb030792469d"} err="failed to get container status \"8ccd098fcf67d43285995c2e9410a210e8509636f32b2af47b3dcb030792469d\": rpc error: code = NotFound desc = could not find container \"8ccd098fcf67d43285995c2e9410a210e8509636f32b2af47b3dcb030792469d\": container with ID starting with 8ccd098fcf67d43285995c2e9410a210e8509636f32b2af47b3dcb030792469d not found: ID does not exist" Nov 02 14:30:13.409977 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.409898 4866 scope.go:122] "RemoveContainer" containerID="87d9c913b7e1fa0faa47f0a777953f201c0c4e429487fa15003149426ea37937" Nov 02 14:30:13.419111 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.419093 4866 scope.go:122] "RemoveContainer" containerID="87d9c913b7e1fa0faa47f0a777953f201c0c4e429487fa15003149426ea37937" Nov 02 14:30:13.419361 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:13.419342 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d9c913b7e1fa0faa47f0a777953f201c0c4e429487fa15003149426ea37937\": container with ID starting with 87d9c913b7e1fa0faa47f0a777953f201c0c4e429487fa15003149426ea37937 not found: ID does not exist" containerID="87d9c913b7e1fa0faa47f0a777953f201c0c4e429487fa15003149426ea37937" Nov 02 14:30:13.419427 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.419369 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d9c913b7e1fa0faa47f0a777953f201c0c4e429487fa15003149426ea37937"} err="failed to get container status \"87d9c913b7e1fa0faa47f0a777953f201c0c4e429487fa15003149426ea37937\": rpc error: code = NotFound desc = could not find container \"87d9c913b7e1fa0faa47f0a777953f201c0c4e429487fa15003149426ea37937\": container with ID starting with 87d9c913b7e1fa0faa47f0a777953f201c0c4e429487fa15003149426ea37937 not found: ID does not exist" Nov 02 14:30:13.453357 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.453322 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl"] Nov 02 14:30:13.460846 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.460817 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5a4525-56bf585947-mzjsl"] Nov 02 14:30:13.492370 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.492334 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf"] Nov 02 14:30:13.496201 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.496173 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5a4525-85499b7f45-5mfwf"] Nov 02 14:30:13.938886 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:13.938855 4866 scope.go:122] "RemoveContainer" containerID="d70003908fff2691596aeb413b69ab078a734893efba62ddcb30f8ebde22960d" Nov 02 14:30:14.043853 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.043820 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z"] Nov 02 14:30:14.044566 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.044545 4866 cpu_manager.go:414] "containerMap: removing container" podUID="bc607e50-88ca-40d8-9a3b-1b1335f74098" containerName="istio-proxy" Nov 02 14:30:14.044566 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.044566 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc607e50-88ca-40d8-9a3b-1b1335f74098" containerName="istio-proxy" Nov 02 14:30:14.044754 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.044591 4866 cpu_manager.go:414] "containerMap: removing container" podUID="c2e5e98d-359e-4417-a474-c3678ab12022" containerName="payload-processing" Nov 02 14:30:14.044754 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.044601 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2e5e98d-359e-4417-a474-c3678ab12022" containerName="payload-processing" Nov 02 14:30:14.044754 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.044624 4866 cpu_manager.go:414] "containerMap: removing container" podUID="427c3914-9b8b-467f-bcb6-ccb99ee098cf" containerName="payload-pre-processing" Nov 02 14:30:14.044754 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.044631 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="427c3914-9b8b-467f-bcb6-ccb99ee098cf" containerName="payload-pre-processing" Nov 02 14:30:14.044962 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.044787 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc607e50-88ca-40d8-9a3b-1b1335f74098" containerName="istio-proxy" Nov 02 14:30:14.044962 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.044804 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2e5e98d-359e-4417-a474-c3678ab12022" containerName="payload-processing" Nov 02 14:30:14.044962 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.044815 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="427c3914-9b8b-467f-bcb6-ccb99ee098cf" containerName="payload-pre-processing" Nov 02 14:30:14.049745 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.049725 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z" Nov 02 14:30:14.069809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.069783 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-928a7a\"" Nov 02 14:30:14.070345 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.070318 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-928a7a-dockercfg-7ldf6\"" Nov 02 14:30:14.070973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.070911 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z"] Nov 02 14:30:14.099357 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.099317 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf6a036-4e2c-4343-bff1-21a55872af0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z\" (UID: \"9bf6a036-4e2c-4343-bff1-21a55872af0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z" Nov 02 14:30:14.099513 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.099424 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n6qk\" (UniqueName: \"kubernetes.io/projected/9bf6a036-4e2c-4343-bff1-21a55872af0a-kube-api-access-2n6qk\") pod \"payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z\" (UID: \"9bf6a036-4e2c-4343-bff1-21a55872af0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z" Nov 02 14:30:14.188969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.188782 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl"] Nov 02 14:30:14.201238 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.201168 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n6qk\" (UniqueName: \"kubernetes.io/projected/9bf6a036-4e2c-4343-bff1-21a55872af0a-kube-api-access-2n6qk\") pod \"payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z\" (UID: \"9bf6a036-4e2c-4343-bff1-21a55872af0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z" Nov 02 14:30:14.201895 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.201759 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf6a036-4e2c-4343-bff1-21a55872af0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z\" (UID: \"9bf6a036-4e2c-4343-bff1-21a55872af0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z" Nov 02 14:30:14.203059 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.203017 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf6a036-4e2c-4343-bff1-21a55872af0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z\" (UID: \"9bf6a036-4e2c-4343-bff1-21a55872af0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z" Nov 02 14:30:14.224971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.219838 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl"] Nov 02 14:30:14.224971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.220046 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl" Nov 02 14:30:14.232212 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.231934 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n6qk\" (UniqueName: \"kubernetes.io/projected/9bf6a036-4e2c-4343-bff1-21a55872af0a-kube-api-access-2n6qk\") pod \"payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z\" (UID: \"9bf6a036-4e2c-4343-bff1-21a55872af0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z" Nov 02 14:30:14.302825 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.302789 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9twnv\" (UniqueName: \"kubernetes.io/projected/505c92f8-9033-4346-a88d-5bde108f70eb-kube-api-access-9twnv\") pod \"payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl\" (UID: \"505c92f8-9033-4346-a88d-5bde108f70eb\") " pod="openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl" Nov 02 14:30:14.303071 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.303048 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/505c92f8-9033-4346-a88d-5bde108f70eb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl\" (UID: \"505c92f8-9033-4346-a88d-5bde108f70eb\") " pod="openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl" Nov 02 14:30:14.363086 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.363046 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z" Nov 02 14:30:14.405376 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.404128 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9twnv\" (UniqueName: \"kubernetes.io/projected/505c92f8-9033-4346-a88d-5bde108f70eb-kube-api-access-9twnv\") pod \"payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl\" (UID: \"505c92f8-9033-4346-a88d-5bde108f70eb\") " pod="openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl" Nov 02 14:30:14.405376 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.404291 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/505c92f8-9033-4346-a88d-5bde108f70eb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl\" (UID: \"505c92f8-9033-4346-a88d-5bde108f70eb\") " pod="openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl" Nov 02 14:30:14.405376 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.405324 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/505c92f8-9033-4346-a88d-5bde108f70eb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl\" (UID: \"505c92f8-9033-4346-a88d-5bde108f70eb\") " pod="openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl" Nov 02 14:30:14.412874 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.412841 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" event={"ID":"de02a12c-740c-45e4-bf68-ec8151644875","Type":"ContainerStarted","Data":"4ab0b021c27cdc6802dc26e4be24e4a8be5b1fecf73aa761cb3d1b006543ea8a"} Nov 02 14:30:14.428423 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.428387 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9twnv\" (UniqueName: \"kubernetes.io/projected/505c92f8-9033-4346-a88d-5bde108f70eb-kube-api-access-9twnv\") pod \"payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl\" (UID: \"505c92f8-9033-4346-a88d-5bde108f70eb\") " pod="openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl" Nov 02 14:30:14.557363 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.557281 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl" Nov 02 14:30:14.675458 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.675409 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z"] Nov 02 14:30:14.679809 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:30:14.679777 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf6a036_4e2c_4343_bff1_21a55872af0a.slice/crio-96ea62b1eb9a3439bbe96900a20009bb1299eacf20dac309bb86715755dbf37b WatchSource:0}: Error finding container 96ea62b1eb9a3439bbe96900a20009bb1299eacf20dac309bb86715755dbf37b: Status 404 returned error can't find the container with id 96ea62b1eb9a3439bbe96900a20009bb1299eacf20dac309bb86715755dbf37b Nov 02 14:30:14.860364 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.860332 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl"] Nov 02 14:30:14.864972 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:30:14.864914 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505c92f8_9033_4346_a88d_5bde108f70eb.slice/crio-7423781e535ae481a01e2cbf0fc5ed28112e6fa521542632f2a7e7bfe8eff20c WatchSource:0}: Error finding container 7423781e535ae481a01e2cbf0fc5ed28112e6fa521542632f2a7e7bfe8eff20c: Status 404 returned error can't find the container with id 7423781e535ae481a01e2cbf0fc5ed28112e6fa521542632f2a7e7bfe8eff20c Nov 02 14:30:14.942489 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.942453 4866 scope.go:122] "RemoveContainer" containerID="a4d24696027bfb0c7f81e42a764e3be7650a5cd23f02db0b83b80e95140a2705" Nov 02 14:30:14.947462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.947437 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="427c3914-9b8b-467f-bcb6-ccb99ee098cf" path="/var/lib/kubelet/pods/427c3914-9b8b-467f-bcb6-ccb99ee098cf/volumes" Nov 02 14:30:14.947835 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:14.947822 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2e5e98d-359e-4417-a474-c3678ab12022" path="/var/lib/kubelet/pods/c2e5e98d-359e-4417-a474-c3678ab12022/volumes" Nov 02 14:30:15.422591 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:15.422499 4866 generic.go:356] "Generic (PLEG): container finished" podID="de02a12c-740c-45e4-bf68-ec8151644875" containerID="4ab0b021c27cdc6802dc26e4be24e4a8be5b1fecf73aa761cb3d1b006543ea8a" exitCode=60 Nov 02 14:30:15.422765 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:15.422703 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" event={"ID":"de02a12c-740c-45e4-bf68-ec8151644875","Type":"ContainerDied","Data":"4ab0b021c27cdc6802dc26e4be24e4a8be5b1fecf73aa761cb3d1b006543ea8a"} Nov 02 14:30:15.422765 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:15.422757 4866 scope.go:122] "RemoveContainer" containerID="d70003908fff2691596aeb413b69ab078a734893efba62ddcb30f8ebde22960d" Nov 02 14:30:15.423273 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:15.423252 4866 scope.go:122] "RemoveContainer" containerID="4ab0b021c27cdc6802dc26e4be24e4a8be5b1fecf73aa761cb3d1b006543ea8a" Nov 02 14:30:15.425379 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:15.425338 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z" event={"ID":"9bf6a036-4e2c-4343-bff1-21a55872af0a","Type":"ContainerStarted","Data":"80457be6e10d1e4f3edd2bb788fd3f78fbd8fabdeafd50d0691c709e76be9ba2"} Nov 02 14:30:15.425472 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:15.425422 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z" event={"ID":"9bf6a036-4e2c-4343-bff1-21a55872af0a","Type":"ContainerStarted","Data":"96ea62b1eb9a3439bbe96900a20009bb1299eacf20dac309bb86715755dbf37b"} Nov 02 14:30:15.425788 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:15.425758 4866 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b_odh-ai-gateway-infra(de02a12c-740c-45e4-bf68-ec8151644875)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" podUID="de02a12c-740c-45e4-bf68-ec8151644875" Nov 02 14:30:15.426160 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:15.426113 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z" Nov 02 14:30:15.427660 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:15.427633 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl" event={"ID":"505c92f8-9033-4346-a88d-5bde108f70eb","Type":"ContainerStarted","Data":"9f527872a72a84e74e806d83f1c6dc0f06626d15f505d10358a4d1b7064671a0"} Nov 02 14:30:15.427770 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:15.427679 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl" event={"ID":"505c92f8-9033-4346-a88d-5bde108f70eb","Type":"ContainerStarted","Data":"7423781e535ae481a01e2cbf0fc5ed28112e6fa521542632f2a7e7bfe8eff20c"} Nov 02 14:30:15.428318 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:15.428285 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl" Nov 02 14:30:15.430345 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:15.430320 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" event={"ID":"3567c919-48d4-4f83-9a1b-32031ddc297f","Type":"ContainerStarted","Data":"8f91ca23123156032e12596c3875ba39ba5dcaf8c69f08fef3f4457acbfb7362"} Nov 02 14:30:15.473268 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:15.473232 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b"] Nov 02 14:30:15.508330 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:15.508275 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" podStartSLOduration=15.508260099 podStartE2EDuration="15.508260099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:15.506376432 +0000 UTC m=+1980.690223770" watchObservedRunningTime="2026-11-02 14:30:15.508260099 +0000 UTC m=+1980.692107435" Nov 02 14:30:15.549593 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:15.549532 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl" podStartSLOduration=1.549509901 podStartE2EDuration="1.549509901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:15.547211485 +0000 UTC m=+1980.731058821" watchObservedRunningTime="2026-11-02 14:30:15.549509901 +0000 UTC m=+1980.733357246" Nov 02 14:30:15.600497 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:15.600440 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z" podStartSLOduration=1.600426246 podStartE2EDuration="1.600426246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:15.597488251 +0000 UTC m=+1980.781335588" watchObservedRunningTime="2026-11-02 14:30:15.600426246 +0000 UTC m=+1980.784273583" Nov 02 14:30:15.938710 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:15.938681 4866 scope.go:122] "RemoveContainer" containerID="a3fd2b2edffc42950cf061642aa67366306aea711b663f923b91e219f5fe2866" Nov 02 14:30:16.438735 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:16.438652 4866 generic.go:356] "Generic (PLEG): container finished" podID="3567c919-48d4-4f83-9a1b-32031ddc297f" containerID="8f91ca23123156032e12596c3875ba39ba5dcaf8c69f08fef3f4457acbfb7362" exitCode=60 Nov 02 14:30:16.439235 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:16.438724 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" event={"ID":"3567c919-48d4-4f83-9a1b-32031ddc297f","Type":"ContainerDied","Data":"8f91ca23123156032e12596c3875ba39ba5dcaf8c69f08fef3f4457acbfb7362"} Nov 02 14:30:16.439235 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:16.438782 4866 scope.go:122] "RemoveContainer" containerID="a4d24696027bfb0c7f81e42a764e3be7650a5cd23f02db0b83b80e95140a2705" Nov 02 14:30:16.439235 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:16.439191 4866 scope.go:122] "RemoveContainer" containerID="8f91ca23123156032e12596c3875ba39ba5dcaf8c69f08fef3f4457acbfb7362" Nov 02 14:30:16.439513 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:16.439489 4866 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8_odh-ai-gateway-infra(3567c919-48d4-4f83-9a1b-32031ddc297f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" podUID="3567c919-48d4-4f83-9a1b-32031ddc297f" Nov 02 14:30:16.441699 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:16.441672 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" event={"ID":"5e672770-0503-4abb-8716-73be17d23f2c","Type":"ContainerStarted","Data":"6dd4643f5a5252e08bb32e2202229c8499b3f9eff77de17e329a60b67a7ee7f6"} Nov 02 14:30:16.525851 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:16.525814 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8"] Nov 02 14:30:16.532541 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:16.532473 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" podStartSLOduration=16.532453565 podStartE2EDuration="16.532453565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:16.531802177 +0000 UTC m=+1981.715649513" watchObservedRunningTime="2026-11-02 14:30:16.532453565 +0000 UTC m=+1981.716300903" Nov 02 14:30:16.607691 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:16.607660 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" Nov 02 14:30:16.636732 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:16.636690 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de02a12c-740c-45e4-bf68-ec8151644875-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de02a12c-740c-45e4-bf68-ec8151644875-maas-api-service-ca\") pod \"de02a12c-740c-45e4-bf68-ec8151644875\" (UID: \"de02a12c-740c-45e4-bf68-ec8151644875\") " Nov 02 14:30:16.638796 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:16.638532 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de02a12c-740c-45e4-bf68-ec8151644875-maas-api-service-ca" pod "de02a12c-740c-45e4-bf68-ec8151644875" (UID: "de02a12c-740c-45e4-bf68-ec8151644875"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:16.642252 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:16.641490 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de02a12c-740c-45e4-bf68-ec8151644875-kube-api-access-8x64r" pod "de02a12c-740c-45e4-bf68-ec8151644875" (UID: "de02a12c-740c-45e4-bf68-ec8151644875"). InnerVolumeSpecName "kube-api-access-8x64r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:16.647066 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:16.636805 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de02a12c-740c-45e4-bf68-ec8151644875-kube-api-access-8x64r\" (UniqueName: \"kubernetes.io/projected/de02a12c-740c-45e4-bf68-ec8151644875-kube-api-access-8x64r\") pod \"de02a12c-740c-45e4-bf68-ec8151644875\" (UID: \"de02a12c-740c-45e4-bf68-ec8151644875\") " Nov 02 14:30:16.647797 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:16.647773 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x64r\" (UniqueName: \"kubernetes.io/projected/de02a12c-740c-45e4-bf68-ec8151644875-kube-api-access-8x64r\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:16.647797 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:16.647797 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de02a12c-740c-45e4-bf68-ec8151644875-maas-api-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:17.455272 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:17.455241 4866 generic.go:356] "Generic (PLEG): container finished" podID="5e672770-0503-4abb-8716-73be17d23f2c" containerID="6dd4643f5a5252e08bb32e2202229c8499b3f9eff77de17e329a60b67a7ee7f6" exitCode=60 Nov 02 14:30:17.455769 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:17.455324 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" event={"ID":"5e672770-0503-4abb-8716-73be17d23f2c","Type":"ContainerDied","Data":"6dd4643f5a5252e08bb32e2202229c8499b3f9eff77de17e329a60b67a7ee7f6"} Nov 02 14:30:17.455769 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:17.455403 4866 scope.go:122] "RemoveContainer" containerID="a3fd2b2edffc42950cf061642aa67366306aea711b663f923b91e219f5fe2866" Nov 02 14:30:17.455883 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:17.455800 4866 scope.go:122] "RemoveContainer" containerID="6dd4643f5a5252e08bb32e2202229c8499b3f9eff77de17e329a60b67a7ee7f6" Nov 02 14:30:17.456152 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:17.456123 4866 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk_odh-ai-gateway-infra(5e672770-0503-4abb-8716-73be17d23f2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" podUID="5e672770-0503-4abb-8716-73be17d23f2c" Nov 02 14:30:17.457120 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:17.456908 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" Nov 02 14:30:17.457120 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:17.456980 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b" event={"ID":"de02a12c-740c-45e4-bf68-ec8151644875","Type":"ContainerDied","Data":"552c96a192a0eebca95ee6e4c6e9b7bf88d64014d86edfaf3fa7c1e3f13cb525"} Nov 02 14:30:17.469160 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:17.469136 4866 scope.go:122] "RemoveContainer" containerID="4ab0b021c27cdc6802dc26e4be24e4a8be5b1fecf73aa761cb3d1b006543ea8a" Nov 02 14:30:17.541801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:17.541766 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b"] Nov 02 14:30:17.547934 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:17.547885 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk"] Nov 02 14:30:17.555386 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:17.552247 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e1c2-29893830-hv75b"] Nov 02 14:30:17.621130 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:17.621103 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" Nov 02 14:30:17.760756 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:17.760666 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3567c919-48d4-4f83-9a1b-32031ddc297f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3567c919-48d4-4f83-9a1b-32031ddc297f-maas-api-service-ca\") pod \"3567c919-48d4-4f83-9a1b-32031ddc297f\" (UID: \"3567c919-48d4-4f83-9a1b-32031ddc297f\") " Nov 02 14:30:17.760937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:17.760817 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3567c919-48d4-4f83-9a1b-32031ddc297f-kube-api-access-7srm4\" (UniqueName: \"kubernetes.io/projected/3567c919-48d4-4f83-9a1b-32031ddc297f-kube-api-access-7srm4\") pod \"3567c919-48d4-4f83-9a1b-32031ddc297f\" (UID: \"3567c919-48d4-4f83-9a1b-32031ddc297f\") " Nov 02 14:30:17.761080 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:17.761055 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3567c919-48d4-4f83-9a1b-32031ddc297f-maas-api-service-ca" pod "3567c919-48d4-4f83-9a1b-32031ddc297f" (UID: "3567c919-48d4-4f83-9a1b-32031ddc297f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:17.761352 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:17.761330 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3567c919-48d4-4f83-9a1b-32031ddc297f-maas-api-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:17.762869 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:17.762847 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3567c919-48d4-4f83-9a1b-32031ddc297f-kube-api-access-7srm4" pod "3567c919-48d4-4f83-9a1b-32031ddc297f" (UID: "3567c919-48d4-4f83-9a1b-32031ddc297f"). InnerVolumeSpecName "kube-api-access-7srm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:17.862524 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:17.862482 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7srm4\" (UniqueName: \"kubernetes.io/projected/3567c919-48d4-4f83-9a1b-32031ddc297f-kube-api-access-7srm4\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:18.466015 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:18.465978 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" event={"ID":"3567c919-48d4-4f83-9a1b-32031ddc297f","Type":"ContainerDied","Data":"4b7b792650c143ecdfd5c92643aeeba9cdfb2e0442c80ac4e114fafda73f1063"} Nov 02 14:30:18.466497 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:18.466018 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8" Nov 02 14:30:18.466497 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:18.466086 4866 scope.go:122] "RemoveContainer" containerID="8f91ca23123156032e12596c3875ba39ba5dcaf8c69f08fef3f4457acbfb7362" Nov 02 14:30:18.575785 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:18.575746 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8"] Nov 02 14:30:18.585238 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:18.585207 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893830-nhlx8"] Nov 02 14:30:18.641277 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:18.641250 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" Nov 02 14:30:18.774531 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:18.774439 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e672770-0503-4abb-8716-73be17d23f2c-kube-api-access-5bhv9\" (UniqueName: \"kubernetes.io/projected/5e672770-0503-4abb-8716-73be17d23f2c-kube-api-access-5bhv9\") pod \"5e672770-0503-4abb-8716-73be17d23f2c\" (UID: \"5e672770-0503-4abb-8716-73be17d23f2c\") " Nov 02 14:30:18.774531 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:18.774521 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e672770-0503-4abb-8716-73be17d23f2c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e672770-0503-4abb-8716-73be17d23f2c-maas-api-service-ca\") pod \"5e672770-0503-4abb-8716-73be17d23f2c\" (UID: \"5e672770-0503-4abb-8716-73be17d23f2c\") " Nov 02 14:30:18.774964 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:18.774921 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e672770-0503-4abb-8716-73be17d23f2c-maas-api-service-ca" pod "5e672770-0503-4abb-8716-73be17d23f2c" (UID: "5e672770-0503-4abb-8716-73be17d23f2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:18.777308 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:18.777270 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e672770-0503-4abb-8716-73be17d23f2c-kube-api-access-5bhv9" pod "5e672770-0503-4abb-8716-73be17d23f2c" (UID: "5e672770-0503-4abb-8716-73be17d23f2c"). InnerVolumeSpecName "kube-api-access-5bhv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:18.876763 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:18.876732 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bhv9\" (UniqueName: \"kubernetes.io/projected/5e672770-0503-4abb-8716-73be17d23f2c-kube-api-access-5bhv9\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:18.876763 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:18.876761 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e672770-0503-4abb-8716-73be17d23f2c-maas-api-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:18.946619 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:18.946588 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3567c919-48d4-4f83-9a1b-32031ddc297f" path="/var/lib/kubelet/pods/3567c919-48d4-4f83-9a1b-32031ddc297f/volumes" Nov 02 14:30:18.947044 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:18.947030 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de02a12c-740c-45e4-bf68-ec8151644875" path="/var/lib/kubelet/pods/de02a12c-740c-45e4-bf68-ec8151644875/volumes" Nov 02 14:30:19.477782 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:19.477745 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" Nov 02 14:30:19.478272 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:19.477750 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk" event={"ID":"5e672770-0503-4abb-8716-73be17d23f2c","Type":"ContainerDied","Data":"76eaf76abb8d178137102a30ad0bdfd5abce40d502c3043e812532f0102ba63b"} Nov 02 14:30:19.478272 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:19.477877 4866 scope.go:122] "RemoveContainer" containerID="6dd4643f5a5252e08bb32e2202229c8499b3f9eff77de17e329a60b67a7ee7f6" Nov 02 14:30:19.546109 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:19.546076 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk"] Nov 02 14:30:19.561046 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:19.561009 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893830-f4ksk"] Nov 02 14:30:20.953283 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:20.953250 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e672770-0503-4abb-8716-73be17d23f2c" path="/var/lib/kubelet/pods/5e672770-0503-4abb-8716-73be17d23f2c/volumes" Nov 02 14:30:21.181799 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.181767 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq"] Nov 02 14:30:21.182419 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182401 4866 cpu_manager.go:414] "containerMap: removing container" podUID="de02a12c-740c-45e4-bf68-ec8151644875" containerName="cleanup" Nov 02 14:30:21.182419 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182419 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de02a12c-740c-45e4-bf68-ec8151644875" containerName="cleanup" Nov 02 14:30:21.182519 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182437 4866 cpu_manager.go:414] "containerMap: removing container" podUID="5e672770-0503-4abb-8716-73be17d23f2c" containerName="cleanup" Nov 02 14:30:21.182519 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182442 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e672770-0503-4abb-8716-73be17d23f2c" containerName="cleanup" Nov 02 14:30:21.182519 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182455 4866 cpu_manager.go:414] "containerMap: removing container" podUID="3567c919-48d4-4f83-9a1b-32031ddc297f" containerName="cleanup" Nov 02 14:30:21.182519 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182464 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3567c919-48d4-4f83-9a1b-32031ddc297f" containerName="cleanup" Nov 02 14:30:21.182519 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182487 4866 cpu_manager.go:414] "containerMap: removing container" podUID="5e672770-0503-4abb-8716-73be17d23f2c" containerName="cleanup" Nov 02 14:30:21.182519 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182492 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e672770-0503-4abb-8716-73be17d23f2c" containerName="cleanup" Nov 02 14:30:21.182519 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182506 4866 cpu_manager.go:414] "containerMap: removing container" podUID="3567c919-48d4-4f83-9a1b-32031ddc297f" containerName="cleanup" Nov 02 14:30:21.182519 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182510 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3567c919-48d4-4f83-9a1b-32031ddc297f" containerName="cleanup" Nov 02 14:30:21.182519 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182518 4866 cpu_manager.go:414] "containerMap: removing container" podUID="5e672770-0503-4abb-8716-73be17d23f2c" containerName="cleanup" Nov 02 14:30:21.182809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182522 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e672770-0503-4abb-8716-73be17d23f2c" containerName="cleanup" Nov 02 14:30:21.182809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182531 4866 cpu_manager.go:414] "containerMap: removing container" podUID="de02a12c-740c-45e4-bf68-ec8151644875" containerName="cleanup" Nov 02 14:30:21.182809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182535 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de02a12c-740c-45e4-bf68-ec8151644875" containerName="cleanup" Nov 02 14:30:21.182809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182623 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="de02a12c-740c-45e4-bf68-ec8151644875" containerName="cleanup" Nov 02 14:30:21.182809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182635 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e672770-0503-4abb-8716-73be17d23f2c" containerName="cleanup" Nov 02 14:30:21.182809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182648 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="3567c919-48d4-4f83-9a1b-32031ddc297f" containerName="cleanup" Nov 02 14:30:21.182809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182656 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="de02a12c-740c-45e4-bf68-ec8151644875" containerName="cleanup" Nov 02 14:30:21.182809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182664 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="3567c919-48d4-4f83-9a1b-32031ddc297f" containerName="cleanup" Nov 02 14:30:21.182809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182671 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="3567c919-48d4-4f83-9a1b-32031ddc297f" containerName="cleanup" Nov 02 14:30:21.182809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.182677 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e672770-0503-4abb-8716-73be17d23f2c" containerName="cleanup" Nov 02 14:30:21.187632 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.187614 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" Nov 02 14:30:21.208755 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.208673 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq"] Nov 02 14:30:21.216256 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.216216 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-778b2bce-dockercfg-rtzpp\"" Nov 02 14:30:21.216256 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.216225 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-778b2bce\"" Nov 02 14:30:21.289264 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.289227 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2"] Nov 02 14:30:21.290459 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.290415 4866 cpu_manager.go:414] "containerMap: removing container" podUID="de02a12c-740c-45e4-bf68-ec8151644875" containerName="cleanup" Nov 02 14:30:21.290459 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.290444 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de02a12c-740c-45e4-bf68-ec8151644875" containerName="cleanup" Nov 02 14:30:21.290690 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.290489 4866 cpu_manager.go:414] "containerMap: removing container" podUID="3567c919-48d4-4f83-9a1b-32031ddc297f" containerName="cleanup" Nov 02 14:30:21.290690 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.290498 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3567c919-48d4-4f83-9a1b-32031ddc297f" containerName="cleanup" Nov 02 14:30:21.290690 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.290649 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e672770-0503-4abb-8716-73be17d23f2c" containerName="cleanup" Nov 02 14:30:21.290690 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.290672 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="de02a12c-740c-45e4-bf68-ec8151644875" containerName="cleanup" Nov 02 14:30:21.309994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.305362 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2"] Nov 02 14:30:21.309994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.305532 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" Nov 02 14:30:21.309994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.308845 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smlkd\" (UniqueName: \"kubernetes.io/projected/325d29d4-150f-4b92-8852-c7df775e9ac1-kube-api-access-smlkd\") pod \"payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq\" (UID: \"325d29d4-150f-4b92-8852-c7df775e9ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" Nov 02 14:30:21.309994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.309883 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325d29d4-150f-4b92-8852-c7df775e9ac1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq\" (UID: \"325d29d4-150f-4b92-8852-c7df775e9ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" Nov 02 14:30:21.412731 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.412692 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65738079-7a3e-4f67-9dbc-0a1b8c067326-plugins-config-volume\") pod \"payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2\" (UID: \"65738079-7a3e-4f67-9dbc-0a1b8c067326\") " pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" Nov 02 14:30:21.412910 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.412814 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325d29d4-150f-4b92-8852-c7df775e9ac1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq\" (UID: \"325d29d4-150f-4b92-8852-c7df775e9ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" Nov 02 14:30:21.412991 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.412911 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjq6q\" (UniqueName: \"kubernetes.io/projected/65738079-7a3e-4f67-9dbc-0a1b8c067326-kube-api-access-cjq6q\") pod \"payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2\" (UID: \"65738079-7a3e-4f67-9dbc-0a1b8c067326\") " pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" Nov 02 14:30:21.413030 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.413018 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smlkd\" (UniqueName: \"kubernetes.io/projected/325d29d4-150f-4b92-8852-c7df775e9ac1-kube-api-access-smlkd\") pod \"payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq\" (UID: \"325d29d4-150f-4b92-8852-c7df775e9ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" Nov 02 14:30:21.413558 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.413537 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325d29d4-150f-4b92-8852-c7df775e9ac1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq\" (UID: \"325d29d4-150f-4b92-8852-c7df775e9ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" Nov 02 14:30:21.445398 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.445360 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smlkd\" (UniqueName: \"kubernetes.io/projected/325d29d4-150f-4b92-8852-c7df775e9ac1-kube-api-access-smlkd\") pod \"payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq\" (UID: \"325d29d4-150f-4b92-8852-c7df775e9ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" Nov 02 14:30:21.507072 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.506987 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" Nov 02 14:30:21.514321 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.514294 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65738079-7a3e-4f67-9dbc-0a1b8c067326-plugins-config-volume\") pod \"payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2\" (UID: \"65738079-7a3e-4f67-9dbc-0a1b8c067326\") " pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" Nov 02 14:30:21.514458 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.514437 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjq6q\" (UniqueName: \"kubernetes.io/projected/65738079-7a3e-4f67-9dbc-0a1b8c067326-kube-api-access-cjq6q\") pod \"payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2\" (UID: \"65738079-7a3e-4f67-9dbc-0a1b8c067326\") " pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" Nov 02 14:30:21.515051 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.515028 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65738079-7a3e-4f67-9dbc-0a1b8c067326-plugins-config-volume\") pod \"payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2\" (UID: \"65738079-7a3e-4f67-9dbc-0a1b8c067326\") " pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" Nov 02 14:30:21.552995 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.552942 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjq6q\" (UniqueName: \"kubernetes.io/projected/65738079-7a3e-4f67-9dbc-0a1b8c067326-kube-api-access-cjq6q\") pod \"payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2\" (UID: \"65738079-7a3e-4f67-9dbc-0a1b8c067326\") " pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" Nov 02 14:30:21.638862 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.638818 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" Nov 02 14:30:21.812485 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.812453 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq"] Nov 02 14:30:21.813699 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:30:21.813655 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325d29d4_150f_4b92_8852_c7df775e9ac1.slice/crio-55e4d870e1a436f79a46edd5033b9062d36b840baa895b2598a1fcce5c203921 WatchSource:0}: Error finding container 55e4d870e1a436f79a46edd5033b9062d36b840baa895b2598a1fcce5c203921: Status 404 returned error can't find the container with id 55e4d870e1a436f79a46edd5033b9062d36b840baa895b2598a1fcce5c203921 Nov 02 14:30:21.896715 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.896681 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb"] Nov 02 14:30:21.896961 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.896920 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" podUID="7697c9b1-291e-4bed-aa54-7e21c05677bd" containerName="maas-api" containerID="cri-o://c9eb8d3fa72f5dca5b6b5c941cff2c0bcb40853f3b6215671ea69a50e63aece9" gracePeriod=30 Nov 02 14:30:21.968278 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:21.968250 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2"] Nov 02 14:30:21.970608 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:30:21.970580 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65738079_7a3e_4f67_9dbc_0a1b8c067326.slice/crio-694ad87251150f0fb1ac907140a88c0d3cebc1c82ed6a9eb871bfb2cd1f2f3a4 WatchSource:0}: Error finding container 694ad87251150f0fb1ac907140a88c0d3cebc1c82ed6a9eb871bfb2cd1f2f3a4: Status 404 returned error can't find the container with id 694ad87251150f0fb1ac907140a88c0d3cebc1c82ed6a9eb871bfb2cd1f2f3a4 Nov 02 14:30:22.195870 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.195516 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" Nov 02 14:30:22.327204 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.327101 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7697c9b1-291e-4bed-aa54-7e21c05677bd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7697c9b1-291e-4bed-aa54-7e21c05677bd-openshift-service-ca\") pod \"7697c9b1-291e-4bed-aa54-7e21c05677bd\" (UID: \"7697c9b1-291e-4bed-aa54-7e21c05677bd\") " Nov 02 14:30:22.327369 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.327266 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7697c9b1-291e-4bed-aa54-7e21c05677bd-kube-api-access-9msn2\" (UniqueName: \"kubernetes.io/projected/7697c9b1-291e-4bed-aa54-7e21c05677bd-kube-api-access-9msn2\") pod \"7697c9b1-291e-4bed-aa54-7e21c05677bd\" (UID: \"7697c9b1-291e-4bed-aa54-7e21c05677bd\") " Nov 02 14:30:22.327369 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.327295 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7697c9b1-291e-4bed-aa54-7e21c05677bd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7697c9b1-291e-4bed-aa54-7e21c05677bd-maas-api-tls\") pod \"7697c9b1-291e-4bed-aa54-7e21c05677bd\" (UID: \"7697c9b1-291e-4bed-aa54-7e21c05677bd\") " Nov 02 14:30:22.327493 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.327389 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7697c9b1-291e-4bed-aa54-7e21c05677bd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7697c9b1-291e-4bed-aa54-7e21c05677bd-metrics-certs\") pod \"7697c9b1-291e-4bed-aa54-7e21c05677bd\" (UID: \"7697c9b1-291e-4bed-aa54-7e21c05677bd\") " Nov 02 14:30:22.327540 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.327493 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7697c9b1-291e-4bed-aa54-7e21c05677bd-openshift-service-ca" pod "7697c9b1-291e-4bed-aa54-7e21c05677bd" (UID: "7697c9b1-291e-4bed-aa54-7e21c05677bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:22.327844 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.327827 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7697c9b1-291e-4bed-aa54-7e21c05677bd-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:22.329431 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.329405 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7697c9b1-291e-4bed-aa54-7e21c05677bd-kube-api-access-9msn2" pod "7697c9b1-291e-4bed-aa54-7e21c05677bd" (UID: "7697c9b1-291e-4bed-aa54-7e21c05677bd"). InnerVolumeSpecName "kube-api-access-9msn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:22.329504 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.329468 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7697c9b1-291e-4bed-aa54-7e21c05677bd-metrics-certs" pod "7697c9b1-291e-4bed-aa54-7e21c05677bd" (UID: "7697c9b1-291e-4bed-aa54-7e21c05677bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:30:22.329543 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.329511 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7697c9b1-291e-4bed-aa54-7e21c05677bd-maas-api-tls" pod "7697c9b1-291e-4bed-aa54-7e21c05677bd" (UID: "7697c9b1-291e-4bed-aa54-7e21c05677bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:30:22.429617 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.429583 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9msn2\" (UniqueName: \"kubernetes.io/projected/7697c9b1-291e-4bed-aa54-7e21c05677bd-kube-api-access-9msn2\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:22.429617 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.429615 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7697c9b1-291e-4bed-aa54-7e21c05677bd-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:22.429617 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.429625 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7697c9b1-291e-4bed-aa54-7e21c05677bd-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:22.502582 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.502547 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" event={"ID":"65738079-7a3e-4f67-9dbc-0a1b8c067326","Type":"ContainerStarted","Data":"e97c4883a9e7d3dfdd3a2857054309bacd6e3344acfdab7e6f21e5b1c8dede7f"} Nov 02 14:30:22.502778 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.502610 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" event={"ID":"65738079-7a3e-4f67-9dbc-0a1b8c067326","Type":"ContainerStarted","Data":"694ad87251150f0fb1ac907140a88c0d3cebc1c82ed6a9eb871bfb2cd1f2f3a4"} Nov 02 14:30:22.502778 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.502754 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" Nov 02 14:30:22.504740 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.504702 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" event={"ID":"325d29d4-150f-4b92-8852-c7df775e9ac1","Type":"ContainerStarted","Data":"73dbace5328b085f0ef58ba3c789f8f335941f0a22421de33d76b3ce23458f76"} Nov 02 14:30:22.504880 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.504755 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" event={"ID":"325d29d4-150f-4b92-8852-c7df775e9ac1","Type":"ContainerStarted","Data":"55e4d870e1a436f79a46edd5033b9062d36b840baa895b2598a1fcce5c203921"} Nov 02 14:30:22.504880 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.504833 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" Nov 02 14:30:22.506520 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.506495 4866 generic.go:356] "Generic (PLEG): container finished" podID="7697c9b1-291e-4bed-aa54-7e21c05677bd" containerID="c9eb8d3fa72f5dca5b6b5c941cff2c0bcb40853f3b6215671ea69a50e63aece9" exitCode=0 Nov 02 14:30:22.506686 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.506528 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" Nov 02 14:30:22.506686 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.506590 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" event={"ID":"7697c9b1-291e-4bed-aa54-7e21c05677bd","Type":"ContainerDied","Data":"c9eb8d3fa72f5dca5b6b5c941cff2c0bcb40853f3b6215671ea69a50e63aece9"} Nov 02 14:30:22.506686 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.506645 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb" event={"ID":"7697c9b1-291e-4bed-aa54-7e21c05677bd","Type":"ContainerDied","Data":"0707af7c81319d91a4b0d77442b86444edced17e843d4b8017d87d642954cbd3"} Nov 02 14:30:22.506686 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.506675 4866 scope.go:122] "RemoveContainer" containerID="c9eb8d3fa72f5dca5b6b5c941cff2c0bcb40853f3b6215671ea69a50e63aece9" Nov 02 14:30:22.518401 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.518252 4866 scope.go:122] "RemoveContainer" containerID="c9eb8d3fa72f5dca5b6b5c941cff2c0bcb40853f3b6215671ea69a50e63aece9" Nov 02 14:30:22.518605 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:22.518570 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9eb8d3fa72f5dca5b6b5c941cff2c0bcb40853f3b6215671ea69a50e63aece9\": container with ID starting with c9eb8d3fa72f5dca5b6b5c941cff2c0bcb40853f3b6215671ea69a50e63aece9 not found: ID does not exist" containerID="c9eb8d3fa72f5dca5b6b5c941cff2c0bcb40853f3b6215671ea69a50e63aece9" Nov 02 14:30:22.518663 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.518605 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9eb8d3fa72f5dca5b6b5c941cff2c0bcb40853f3b6215671ea69a50e63aece9"} err="failed to get container status \"c9eb8d3fa72f5dca5b6b5c941cff2c0bcb40853f3b6215671ea69a50e63aece9\": rpc error: code = NotFound desc = could not find container \"c9eb8d3fa72f5dca5b6b5c941cff2c0bcb40853f3b6215671ea69a50e63aece9\": container with ID starting with c9eb8d3fa72f5dca5b6b5c941cff2c0bcb40853f3b6215671ea69a50e63aece9 not found: ID does not exist" Nov 02 14:30:22.571879 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.571822 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" podStartSLOduration=1.571805084 podStartE2EDuration="1.571805084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:22.568558415 +0000 UTC m=+1987.752405754" watchObservedRunningTime="2026-11-02 14:30:22.571805084 +0000 UTC m=+1987.755652420" Nov 02 14:30:22.640730 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.640627 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" podStartSLOduration=1.6406132549999999 podStartE2EDuration="1.640613255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:22.640022321 +0000 UTC m=+1987.823869659" watchObservedRunningTime="2026-11-02 14:30:22.640613255 +0000 UTC m=+1987.824460592" Nov 02 14:30:22.684106 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.684072 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb"] Nov 02 14:30:22.690538 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.690508 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5a4525-68869c56dd-jltfb"] Nov 02 14:30:22.947011 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:22.946942 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7697c9b1-291e-4bed-aa54-7e21c05677bd" path="/var/lib/kubelet/pods/7697c9b1-291e-4bed-aa54-7e21c05677bd/volumes" Nov 02 14:30:25.506852 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:25.506821 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4"] Nov 02 14:30:25.507294 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:25.507035 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" podUID="c91b5d92-2a59-46c7-ac62-f398a392a8ff" containerName="istio-proxy" containerID="cri-o://f1e87085bce0956eca1d982923b8b4001babf71e6ac41dcefe1143e0242f8350" gracePeriod=30 Nov 02 14:30:26.449658 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:26.449629 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z" Nov 02 14:30:27.463356 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:27.463323 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl" Nov 02 14:30:30.777653 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.777585 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:30:30.946234 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.946201 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-credential-socket\") pod \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " Nov 02 14:30:30.946409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.946280 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-podinfo\") pod \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " Nov 02 14:30:30.946409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.946333 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c91b5d92-2a59-46c7-ac62-f398a392a8ff-kube-api-access-fph7d\" (UniqueName: \"kubernetes.io/projected/c91b5d92-2a59-46c7-ac62-f398a392a8ff-kube-api-access-fph7d\") pod \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " Nov 02 14:30:30.946409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.946358 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istiod-ca-cert\") pod \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " Nov 02 14:30:30.946409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.946406 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-workload-certs\") pod \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " Nov 02 14:30:30.946640 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.946442 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-credential-socket" pod "c91b5d92-2a59-46c7-ac62-f398a392a8ff" (UID: "c91b5d92-2a59-46c7-ac62-f398a392a8ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:30:30.946714 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.946691 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-workload-socket\") pod \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " Nov 02 14:30:30.946714 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.946705 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-workload-certs" pod "c91b5d92-2a59-46c7-ac62-f398a392a8ff" (UID: "c91b5d92-2a59-46c7-ac62-f398a392a8ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:30:30.946842 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.946817 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istiod-ca-cert" pod "c91b5d92-2a59-46c7-ac62-f398a392a8ff" (UID: "c91b5d92-2a59-46c7-ac62-f398a392a8ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:30.946900 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.946827 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-token\" (UniqueName: \"kubernetes.io/projected/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-token\") pod \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " Nov 02 14:30:30.946994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.946976 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-workload-socket" pod "c91b5d92-2a59-46c7-ac62-f398a392a8ff" (UID: "c91b5d92-2a59-46c7-ac62-f398a392a8ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:30:30.947060 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.947047 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-envoy\") pod \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " Nov 02 14:30:30.947503 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.947083 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-data\") pod \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\" (UID: \"c91b5d92-2a59-46c7-ac62-f398a392a8ff\") " Nov 02 14:30:30.947503 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.947396 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-data" pod "c91b5d92-2a59-46c7-ac62-f398a392a8ff" (UID: "c91b5d92-2a59-46c7-ac62-f398a392a8ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:30:30.947663 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.947611 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:30.947663 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.947635 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:30.947663 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.947652 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:30.947847 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.947668 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:30.947847 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.947682 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:30.949127 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.949100 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-podinfo" pod "c91b5d92-2a59-46c7-ac62-f398a392a8ff" (UID: "c91b5d92-2a59-46c7-ac62-f398a392a8ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:30:30.949238 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.949139 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91b5d92-2a59-46c7-ac62-f398a392a8ff-kube-api-access-fph7d" pod "c91b5d92-2a59-46c7-ac62-f398a392a8ff" (UID: "c91b5d92-2a59-46c7-ac62-f398a392a8ff"). InnerVolumeSpecName "kube-api-access-fph7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:30.949308 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.949259 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-token" pod "c91b5d92-2a59-46c7-ac62-f398a392a8ff" (UID: "c91b5d92-2a59-46c7-ac62-f398a392a8ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:30.949366 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:30.949323 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-envoy" pod "c91b5d92-2a59-46c7-ac62-f398a392a8ff" (UID: "c91b5d92-2a59-46c7-ac62-f398a392a8ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:30:31.048680 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:31.048602 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:31.048680 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:31.048632 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:31.048680 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:31.048642 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c91b5d92-2a59-46c7-ac62-f398a392a8ff-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:31.048680 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:31.048651 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fph7d\" (UniqueName: \"kubernetes.io/projected/c91b5d92-2a59-46c7-ac62-f398a392a8ff-kube-api-access-fph7d\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:30:31.585667 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:31.585635 4866 generic.go:356] "Generic (PLEG): container finished" podID="c91b5d92-2a59-46c7-ac62-f398a392a8ff" containerID="f1e87085bce0956eca1d982923b8b4001babf71e6ac41dcefe1143e0242f8350" exitCode=0 Nov 02 14:30:31.585840 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:31.585700 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" Nov 02 14:30:31.585840 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:31.585719 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" event={"ID":"c91b5d92-2a59-46c7-ac62-f398a392a8ff","Type":"ContainerDied","Data":"f1e87085bce0956eca1d982923b8b4001babf71e6ac41dcefe1143e0242f8350"} Nov 02 14:30:31.585840 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:31.585774 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4" event={"ID":"c91b5d92-2a59-46c7-ac62-f398a392a8ff","Type":"ContainerDied","Data":"b8e3c9ac53b2cbcb27dbcc81048b83ab56705df8c5687c3e69b835196b59e57e"} Nov 02 14:30:31.585840 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:31.585796 4866 scope.go:122] "RemoveContainer" containerID="f1e87085bce0956eca1d982923b8b4001babf71e6ac41dcefe1143e0242f8350" Nov 02 14:30:31.596526 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:31.596510 4866 scope.go:122] "RemoveContainer" containerID="d63f03129bbf648a49d5216dbedb2e3213b6348960d81522142b0b3b31842d0b" Nov 02 14:30:31.605763 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:31.605744 4866 scope.go:122] "RemoveContainer" containerID="f1e87085bce0956eca1d982923b8b4001babf71e6ac41dcefe1143e0242f8350" Nov 02 14:30:31.606054 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:31.606033 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e87085bce0956eca1d982923b8b4001babf71e6ac41dcefe1143e0242f8350\": container with ID starting with f1e87085bce0956eca1d982923b8b4001babf71e6ac41dcefe1143e0242f8350 not found: ID does not exist" containerID="f1e87085bce0956eca1d982923b8b4001babf71e6ac41dcefe1143e0242f8350" Nov 02 14:30:31.606095 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:31.606064 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e87085bce0956eca1d982923b8b4001babf71e6ac41dcefe1143e0242f8350"} err="failed to get container status \"f1e87085bce0956eca1d982923b8b4001babf71e6ac41dcefe1143e0242f8350\": rpc error: code = NotFound desc = could not find container \"f1e87085bce0956eca1d982923b8b4001babf71e6ac41dcefe1143e0242f8350\": container with ID starting with f1e87085bce0956eca1d982923b8b4001babf71e6ac41dcefe1143e0242f8350 not found: ID does not exist" Nov 02 14:30:31.606095 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:31.606085 4866 scope.go:122] "RemoveContainer" containerID="d63f03129bbf648a49d5216dbedb2e3213b6348960d81522142b0b3b31842d0b" Nov 02 14:30:31.606344 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:31.606326 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d63f03129bbf648a49d5216dbedb2e3213b6348960d81522142b0b3b31842d0b\": container with ID starting with d63f03129bbf648a49d5216dbedb2e3213b6348960d81522142b0b3b31842d0b not found: ID does not exist" containerID="d63f03129bbf648a49d5216dbedb2e3213b6348960d81522142b0b3b31842d0b" Nov 02 14:30:31.606396 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:31.606349 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d63f03129bbf648a49d5216dbedb2e3213b6348960d81522142b0b3b31842d0b"} err="failed to get container status \"d63f03129bbf648a49d5216dbedb2e3213b6348960d81522142b0b3b31842d0b\": rpc error: code = NotFound desc = could not find container \"d63f03129bbf648a49d5216dbedb2e3213b6348960d81522142b0b3b31842d0b\": container with ID starting with d63f03129bbf648a49d5216dbedb2e3213b6348960d81522142b0b3b31842d0b not found: ID does not exist" Nov 02 14:30:31.650979 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:31.650926 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4"] Nov 02 14:30:31.662189 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:31.662153 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-5a4525-openshift-default-cd8fbcb69-t6tv4"] Nov 02 14:30:32.947930 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:32.947899 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c91b5d92-2a59-46c7-ac62-f398a392a8ff" path="/var/lib/kubelet/pods/c91b5d92-2a59-46c7-ac62-f398a392a8ff/volumes" Nov 02 14:30:33.517111 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:33.517079 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" Nov 02 14:30:33.517293 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:33.517160 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" Nov 02 14:30:36.011122 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:36.011081 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:30:41.071560 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.071524 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242"] Nov 02 14:30:41.072330 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.072309 4866 cpu_manager.go:414] "containerMap: removing container" podUID="7697c9b1-291e-4bed-aa54-7e21c05677bd" containerName="maas-api" Nov 02 14:30:41.072424 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.072331 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7697c9b1-291e-4bed-aa54-7e21c05677bd" containerName="maas-api" Nov 02 14:30:41.072424 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.072348 4866 cpu_manager.go:414] "containerMap: removing container" podUID="c91b5d92-2a59-46c7-ac62-f398a392a8ff" containerName="istio-proxy" Nov 02 14:30:41.072424 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.072355 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c91b5d92-2a59-46c7-ac62-f398a392a8ff" containerName="istio-proxy" Nov 02 14:30:41.072424 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.072374 4866 cpu_manager.go:414] "containerMap: removing container" podUID="c91b5d92-2a59-46c7-ac62-f398a392a8ff" containerName="istio-proxy" Nov 02 14:30:41.072424 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.072382 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c91b5d92-2a59-46c7-ac62-f398a392a8ff" containerName="istio-proxy" Nov 02 14:30:41.073033 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.072520 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="7697c9b1-291e-4bed-aa54-7e21c05677bd" containerName="maas-api" Nov 02 14:30:41.073033 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.072535 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="c91b5d92-2a59-46c7-ac62-f398a392a8ff" containerName="istio-proxy" Nov 02 14:30:41.073033 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.072545 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="c91b5d92-2a59-46c7-ac62-f398a392a8ff" containerName="istio-proxy" Nov 02 14:30:41.076026 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.076006 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" Nov 02 14:30:41.090931 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.090911 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-928a7a-serving-cert\"" Nov 02 14:30:41.106634 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.106601 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242"] Nov 02 14:30:41.170136 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.170099 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plrf6\" (UniqueName: \"kubernetes.io/projected/e62231f6-df12-4423-9b82-04db2235d365-kube-api-access-plrf6\") pod \"maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242\" (UID: \"e62231f6-df12-4423-9b82-04db2235d365\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" Nov 02 14:30:41.170333 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.170301 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62231f6-df12-4423-9b82-04db2235d365-openshift-service-ca\") pod \"maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242\" (UID: \"e62231f6-df12-4423-9b82-04db2235d365\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" Nov 02 14:30:41.170416 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.170356 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62231f6-df12-4423-9b82-04db2235d365-metrics-certs\") pod \"maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242\" (UID: \"e62231f6-df12-4423-9b82-04db2235d365\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" Nov 02 14:30:41.170462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.170432 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62231f6-df12-4423-9b82-04db2235d365-maas-api-tls\") pod \"maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242\" (UID: \"e62231f6-df12-4423-9b82-04db2235d365\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" Nov 02 14:30:41.271646 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.271607 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plrf6\" (UniqueName: \"kubernetes.io/projected/e62231f6-df12-4423-9b82-04db2235d365-kube-api-access-plrf6\") pod \"maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242\" (UID: \"e62231f6-df12-4423-9b82-04db2235d365\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" Nov 02 14:30:41.271831 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.271749 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62231f6-df12-4423-9b82-04db2235d365-openshift-service-ca\") pod \"maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242\" (UID: \"e62231f6-df12-4423-9b82-04db2235d365\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" Nov 02 14:30:41.271831 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.271788 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62231f6-df12-4423-9b82-04db2235d365-metrics-certs\") pod \"maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242\" (UID: \"e62231f6-df12-4423-9b82-04db2235d365\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" Nov 02 14:30:41.271918 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.271858 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62231f6-df12-4423-9b82-04db2235d365-maas-api-tls\") pod \"maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242\" (UID: \"e62231f6-df12-4423-9b82-04db2235d365\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" Nov 02 14:30:41.272479 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.272443 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62231f6-df12-4423-9b82-04db2235d365-openshift-service-ca\") pod \"maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242\" (UID: \"e62231f6-df12-4423-9b82-04db2235d365\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" Nov 02 14:30:41.274427 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.274403 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62231f6-df12-4423-9b82-04db2235d365-maas-api-tls\") pod \"maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242\" (UID: \"e62231f6-df12-4423-9b82-04db2235d365\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" Nov 02 14:30:41.274533 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.274516 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62231f6-df12-4423-9b82-04db2235d365-metrics-certs\") pod \"maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242\" (UID: \"e62231f6-df12-4423-9b82-04db2235d365\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" Nov 02 14:30:41.294833 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.294800 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plrf6\" (UniqueName: \"kubernetes.io/projected/e62231f6-df12-4423-9b82-04db2235d365-kube-api-access-plrf6\") pod \"maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242\" (UID: \"e62231f6-df12-4423-9b82-04db2235d365\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" Nov 02 14:30:41.389480 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.389402 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" Nov 02 14:30:41.713293 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:41.713264 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242"] Nov 02 14:30:41.714148 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:30:41.714117 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62231f6_df12_4423_9b82_04db2235d365.slice/crio-3d9ac64c9b56035102d54ef74e75b7498fc1eb77de2122efb8ed1dd417e881f7 WatchSource:0}: Error finding container 3d9ac64c9b56035102d54ef74e75b7498fc1eb77de2122efb8ed1dd417e881f7: Status 404 returned error can't find the container with id 3d9ac64c9b56035102d54ef74e75b7498fc1eb77de2122efb8ed1dd417e881f7 Nov 02 14:30:42.658792 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:42.658760 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" event={"ID":"e62231f6-df12-4423-9b82-04db2235d365","Type":"ContainerStarted","Data":"81f9ea67633e8c3c8b2c782d874fb3d32e6f44d83659a515df30a83e81d1d294"} Nov 02 14:30:42.659243 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:42.658815 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" event={"ID":"e62231f6-df12-4423-9b82-04db2235d365","Type":"ContainerStarted","Data":"3d9ac64c9b56035102d54ef74e75b7498fc1eb77de2122efb8ed1dd417e881f7"} Nov 02 14:30:42.659243 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:42.658896 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" Nov 02 14:30:42.707081 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:42.707031 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" podStartSLOduration=1.37574672 podStartE2EDuration="1.707016108s" totalImagesPullingTime="331.269388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:42.706423363 +0000 UTC m=+2007.890270700" watchObservedRunningTime="2026-11-02 14:30:42.707016108 +0000 UTC m=+2007.890863445" Nov 02 14:30:48.265375 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:48.265340 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w"] Nov 02 14:30:48.270897 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:48.270873 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" Nov 02 14:30:48.291228 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:48.291193 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w"] Nov 02 14:30:48.299994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:48.299960 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-778b2bce-serving-cert\"" Nov 02 14:30:48.360235 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:48.360197 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186b144a-c571-47a5-8ca0-264b72fcaea4-metrics-certs\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w\" (UID: \"186b144a-c571-47a5-8ca0-264b72fcaea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" Nov 02 14:30:48.360450 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:48.360260 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186b144a-c571-47a5-8ca0-264b72fcaea4-openshift-service-ca\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w\" (UID: \"186b144a-c571-47a5-8ca0-264b72fcaea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" Nov 02 14:30:48.360450 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:48.360371 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186b144a-c571-47a5-8ca0-264b72fcaea4-maas-api-tls\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w\" (UID: \"186b144a-c571-47a5-8ca0-264b72fcaea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" Nov 02 14:30:48.360552 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:48.360474 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df8nb\" (UniqueName: \"kubernetes.io/projected/186b144a-c571-47a5-8ca0-264b72fcaea4-kube-api-access-df8nb\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w\" (UID: \"186b144a-c571-47a5-8ca0-264b72fcaea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" Nov 02 14:30:48.461930 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:48.461883 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186b144a-c571-47a5-8ca0-264b72fcaea4-openshift-service-ca\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w\" (UID: \"186b144a-c571-47a5-8ca0-264b72fcaea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" Nov 02 14:30:48.462171 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:48.461974 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186b144a-c571-47a5-8ca0-264b72fcaea4-maas-api-tls\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w\" (UID: \"186b144a-c571-47a5-8ca0-264b72fcaea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" Nov 02 14:30:48.462171 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:48.462042 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df8nb\" (UniqueName: \"kubernetes.io/projected/186b144a-c571-47a5-8ca0-264b72fcaea4-kube-api-access-df8nb\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w\" (UID: \"186b144a-c571-47a5-8ca0-264b72fcaea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" Nov 02 14:30:48.462323 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:48.462274 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186b144a-c571-47a5-8ca0-264b72fcaea4-metrics-certs\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w\" (UID: \"186b144a-c571-47a5-8ca0-264b72fcaea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" Nov 02 14:30:48.463337 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:48.463097 4866 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-serving-cert: secret "maas-api-e2e-mt-778b2bce-serving-cert" not found Nov 02 14:30:48.463337 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:30:48.463185 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/186b144a-c571-47a5-8ca0-264b72fcaea4-maas-api-tls podName:186b144a-c571-47a5-8ca0-264b72fcaea4 nodeName:}" failed. No retries permitted until 2026-11-02 14:30:48.963162913 +0000 UTC m=+2014.147010243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/186b144a-c571-47a5-8ca0-264b72fcaea4-maas-api-tls") pod "maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" (UID: "186b144a-c571-47a5-8ca0-264b72fcaea4") : secret "maas-api-e2e-mt-778b2bce-serving-cert" not found Nov 02 14:30:48.464209 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:48.464169 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186b144a-c571-47a5-8ca0-264b72fcaea4-openshift-service-ca\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w\" (UID: \"186b144a-c571-47a5-8ca0-264b72fcaea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" Nov 02 14:30:48.467537 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:48.467509 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186b144a-c571-47a5-8ca0-264b72fcaea4-metrics-certs\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w\" (UID: \"186b144a-c571-47a5-8ca0-264b72fcaea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" Nov 02 14:30:48.497997 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:48.495584 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df8nb\" (UniqueName: \"kubernetes.io/projected/186b144a-c571-47a5-8ca0-264b72fcaea4-kube-api-access-df8nb\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w\" (UID: \"186b144a-c571-47a5-8ca0-264b72fcaea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" Nov 02 14:30:48.670906 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:48.670827 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" Nov 02 14:30:48.969047 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:48.969007 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186b144a-c571-47a5-8ca0-264b72fcaea4-maas-api-tls\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w\" (UID: \"186b144a-c571-47a5-8ca0-264b72fcaea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" Nov 02 14:30:48.971564 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:48.971537 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186b144a-c571-47a5-8ca0-264b72fcaea4-maas-api-tls\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w\" (UID: \"186b144a-c571-47a5-8ca0-264b72fcaea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" Nov 02 14:30:49.186296 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:49.186011 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" Nov 02 14:30:49.531378 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:49.528880 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w"] Nov 02 14:30:49.561884 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:30:49.561819 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186b144a_c571_47a5_8ca0_264b72fcaea4.slice/crio-4de41038c023099468bf177dad440f1d59959d1aa9ef2218daa047ac1861f7cf WatchSource:0}: Error finding container 4de41038c023099468bf177dad440f1d59959d1aa9ef2218daa047ac1861f7cf: Status 404 returned error can't find the container with id 4de41038c023099468bf177dad440f1d59959d1aa9ef2218daa047ac1861f7cf Nov 02 14:30:49.709897 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:49.709861 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" event={"ID":"186b144a-c571-47a5-8ca0-264b72fcaea4","Type":"ContainerStarted","Data":"4de41038c023099468bf177dad440f1d59959d1aa9ef2218daa047ac1861f7cf"} Nov 02 14:30:50.718521 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:50.718489 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" event={"ID":"186b144a-c571-47a5-8ca0-264b72fcaea4","Type":"ContainerStarted","Data":"1374fdacd06cd25e1f36197f7e0ca91c50193c5e4dc1fc31a2a9559a82596164"} Nov 02 14:30:50.718937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:50.718655 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" Nov 02 14:30:50.761662 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:50.761614 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" podStartSLOduration=2.2909435719999998 podStartE2EDuration="2.761596204s" totalImagesPullingTime="470.652632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:50.761428028 +0000 UTC m=+2015.945275365" watchObservedRunningTime="2026-11-02 14:30:50.761596204 +0000 UTC m=+2015.945443540" Nov 02 14:30:56.730232 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:30:56.730202 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" Nov 02 14:31:06.010208 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:31:06.010122 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:31:07.335708 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:07.335669 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5"] Nov 02 14:31:07.341137 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:07.341120 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" Nov 02 14:31:07.361542 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:07.361513 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-e07583\"" Nov 02 14:31:07.376781 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:07.376743 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5"] Nov 02 14:31:07.480518 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:07.480476 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzdqk\" (UniqueName: \"kubernetes.io/projected/2d243c72-dc2f-458e-8f7b-23f89144e7bf-kube-api-access-rzdqk\") pod \"payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5\" (UID: \"2d243c72-dc2f-458e-8f7b-23f89144e7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" Nov 02 14:31:07.480738 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:07.480689 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d243c72-dc2f-458e-8f7b-23f89144e7bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5\" (UID: \"2d243c72-dc2f-458e-8f7b-23f89144e7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" Nov 02 14:31:07.582496 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:07.582463 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzdqk\" (UniqueName: \"kubernetes.io/projected/2d243c72-dc2f-458e-8f7b-23f89144e7bf-kube-api-access-rzdqk\") pod \"payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5\" (UID: \"2d243c72-dc2f-458e-8f7b-23f89144e7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" Nov 02 14:31:07.582685 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:07.582607 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d243c72-dc2f-458e-8f7b-23f89144e7bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5\" (UID: \"2d243c72-dc2f-458e-8f7b-23f89144e7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" Nov 02 14:31:07.583333 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:07.583311 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d243c72-dc2f-458e-8f7b-23f89144e7bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5\" (UID: \"2d243c72-dc2f-458e-8f7b-23f89144e7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" Nov 02 14:31:07.608043 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:07.608011 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzdqk\" (UniqueName: \"kubernetes.io/projected/2d243c72-dc2f-458e-8f7b-23f89144e7bf-kube-api-access-rzdqk\") pod \"payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5\" (UID: \"2d243c72-dc2f-458e-8f7b-23f89144e7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" Nov 02 14:31:07.655764 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:07.655727 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" Nov 02 14:31:07.961368 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:07.961338 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5"] Nov 02 14:31:07.962037 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:31:07.962008 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d243c72_dc2f_458e_8f7b_23f89144e7bf.slice/crio-59288b47122cc17e549799eaf7c1dd6a899645997b8617e8a4249043ef53a199 WatchSource:0}: Error finding container 59288b47122cc17e549799eaf7c1dd6a899645997b8617e8a4249043ef53a199: Status 404 returned error can't find the container with id 59288b47122cc17e549799eaf7c1dd6a899645997b8617e8a4249043ef53a199 Nov 02 14:31:08.846525 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:08.846490 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" event={"ID":"2d243c72-dc2f-458e-8f7b-23f89144e7bf","Type":"ContainerStarted","Data":"4621448a13b0db718dd710e88f30e200a2c16a31fd09008b226711c9b666acab"} Nov 02 14:31:08.847026 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:08.846542 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" event={"ID":"2d243c72-dc2f-458e-8f7b-23f89144e7bf","Type":"ContainerStarted","Data":"59288b47122cc17e549799eaf7c1dd6a899645997b8617e8a4249043ef53a199"} Nov 02 14:31:08.847026 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:08.846685 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" Nov 02 14:31:08.893340 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:08.893158 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" podStartSLOduration=1.893140556 podStartE2EDuration="1.893140556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:31:08.891001653 +0000 UTC m=+2034.074848991" watchObservedRunningTime="2026-11-02 14:31:08.893140556 +0000 UTC m=+2034.076987892" Nov 02 14:31:19.856086 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:19.856055 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" Nov 02 14:31:30.722894 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:30.722856 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx"] Nov 02 14:31:35.433588 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:35.433552 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk"] Nov 02 14:31:35.434110 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:35.433784 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" podUID="1cf97eaa-077b-4652-8907-acd8bfbadade" containerName="maas-api" containerID="cri-o://21178d4aff6b0e12402bfee2d69f9bc464867eb4928f5ac617b0c2ec7d17b6b3" gracePeriod=30 Nov 02 14:31:35.723807 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:35.723769 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx" podUID="41e0f03e-11af-465d-aef2-8c3ef52a06bf" containerName="payload-processing" containerID="cri-o://df53f7a442a806c5aa847f8f7042db5fa3e1799fff160df74c1ab26cf9f70c87" gracePeriod=25 Nov 02 14:31:35.851969 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:31:35.844145 4866 logging.go:55] [core] [Channel #2486 SubChannel #2487]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.115:50052", ServerName: "10.132.0.115:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.115:50052: connect: connection refused" Nov 02 14:31:35.851969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:35.851345 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" Nov 02 14:31:35.949861 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:35.949824 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cf97eaa-077b-4652-8907-acd8bfbadade-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cf97eaa-077b-4652-8907-acd8bfbadade-metrics-certs\") pod \"1cf97eaa-077b-4652-8907-acd8bfbadade\" (UID: \"1cf97eaa-077b-4652-8907-acd8bfbadade\") " Nov 02 14:31:35.950148 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:35.950132 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cf97eaa-077b-4652-8907-acd8bfbadade-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cf97eaa-077b-4652-8907-acd8bfbadade-maas-api-tls\") pod \"1cf97eaa-077b-4652-8907-acd8bfbadade\" (UID: \"1cf97eaa-077b-4652-8907-acd8bfbadade\") " Nov 02 14:31:35.950298 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:35.950274 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf97eaa-077b-4652-8907-acd8bfbadade-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf97eaa-077b-4652-8907-acd8bfbadade-openshift-service-ca\") pod \"1cf97eaa-077b-4652-8907-acd8bfbadade\" (UID: \"1cf97eaa-077b-4652-8907-acd8bfbadade\") " Nov 02 14:31:35.950445 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:35.950433 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf97eaa-077b-4652-8907-acd8bfbadade-kube-api-access-6hss8\" (UniqueName: \"kubernetes.io/projected/1cf97eaa-077b-4652-8907-acd8bfbadade-kube-api-access-6hss8\") pod \"1cf97eaa-077b-4652-8907-acd8bfbadade\" (UID: \"1cf97eaa-077b-4652-8907-acd8bfbadade\") " Nov 02 14:31:35.956930 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:35.956530 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cf97eaa-077b-4652-8907-acd8bfbadade-metrics-certs" pod "1cf97eaa-077b-4652-8907-acd8bfbadade" (UID: "1cf97eaa-077b-4652-8907-acd8bfbadade"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:31:35.958024 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:35.957992 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf97eaa-077b-4652-8907-acd8bfbadade-openshift-service-ca" pod "1cf97eaa-077b-4652-8907-acd8bfbadade" (UID: "1cf97eaa-077b-4652-8907-acd8bfbadade"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:31:35.958346 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:35.958321 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf97eaa-077b-4652-8907-acd8bfbadade-kube-api-access-6hss8" pod "1cf97eaa-077b-4652-8907-acd8bfbadade" (UID: "1cf97eaa-077b-4652-8907-acd8bfbadade"). InnerVolumeSpecName "kube-api-access-6hss8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:31:35.959789 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:35.959763 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cf97eaa-077b-4652-8907-acd8bfbadade-maas-api-tls" pod "1cf97eaa-077b-4652-8907-acd8bfbadade" (UID: "1cf97eaa-077b-4652-8907-acd8bfbadade"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:31:36.025991 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:31:36.025915 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:31:36.049673 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.049632 4866 generic.go:356] "Generic (PLEG): container finished" podID="41e0f03e-11af-465d-aef2-8c3ef52a06bf" containerID="df53f7a442a806c5aa847f8f7042db5fa3e1799fff160df74c1ab26cf9f70c87" exitCode=0 Nov 02 14:31:36.049842 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.049685 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx" event={"ID":"41e0f03e-11af-465d-aef2-8c3ef52a06bf","Type":"ContainerDied","Data":"df53f7a442a806c5aa847f8f7042db5fa3e1799fff160df74c1ab26cf9f70c87"} Nov 02 14:31:36.051315 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.051294 4866 generic.go:356] "Generic (PLEG): container finished" podID="1cf97eaa-077b-4652-8907-acd8bfbadade" containerID="21178d4aff6b0e12402bfee2d69f9bc464867eb4928f5ac617b0c2ec7d17b6b3" exitCode=0 Nov 02 14:31:36.051455 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.051363 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" Nov 02 14:31:36.051455 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.051364 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" event={"ID":"1cf97eaa-077b-4652-8907-acd8bfbadade","Type":"ContainerDied","Data":"21178d4aff6b0e12402bfee2d69f9bc464867eb4928f5ac617b0c2ec7d17b6b3"} Nov 02 14:31:36.051556 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.051481 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk" event={"ID":"1cf97eaa-077b-4652-8907-acd8bfbadade","Type":"ContainerDied","Data":"5afa1babf7de10f73e93cc7c156735e90d72292752f8a83a16fa93dbcc456f40"} Nov 02 14:31:36.051556 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.051507 4866 scope.go:122] "RemoveContainer" containerID="21178d4aff6b0e12402bfee2d69f9bc464867eb4928f5ac617b0c2ec7d17b6b3" Nov 02 14:31:36.052134 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.052099 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cf97eaa-077b-4652-8907-acd8bfbadade-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:31:36.052134 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.052123 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cf97eaa-077b-4652-8907-acd8bfbadade-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:31:36.052912 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.052138 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf97eaa-077b-4652-8907-acd8bfbadade-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:31:36.052912 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.052153 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hss8\" (UniqueName: \"kubernetes.io/projected/1cf97eaa-077b-4652-8907-acd8bfbadade-kube-api-access-6hss8\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:31:36.069063 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.068436 4866 scope.go:122] "RemoveContainer" containerID="21178d4aff6b0e12402bfee2d69f9bc464867eb4928f5ac617b0c2ec7d17b6b3" Nov 02 14:31:36.069063 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:31:36.068912 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21178d4aff6b0e12402bfee2d69f9bc464867eb4928f5ac617b0c2ec7d17b6b3\": container with ID starting with 21178d4aff6b0e12402bfee2d69f9bc464867eb4928f5ac617b0c2ec7d17b6b3 not found: ID does not exist" containerID="21178d4aff6b0e12402bfee2d69f9bc464867eb4928f5ac617b0c2ec7d17b6b3" Nov 02 14:31:36.069063 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.068989 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21178d4aff6b0e12402bfee2d69f9bc464867eb4928f5ac617b0c2ec7d17b6b3"} err="failed to get container status \"21178d4aff6b0e12402bfee2d69f9bc464867eb4928f5ac617b0c2ec7d17b6b3\": rpc error: code = NotFound desc = could not find container \"21178d4aff6b0e12402bfee2d69f9bc464867eb4928f5ac617b0c2ec7d17b6b3\": container with ID starting with 21178d4aff6b0e12402bfee2d69f9bc464867eb4928f5ac617b0c2ec7d17b6b3 not found: ID does not exist" Nov 02 14:31:36.081659 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.081634 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx" Nov 02 14:31:36.089995 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.089963 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2"] Nov 02 14:31:36.090593 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.090577 4866 cpu_manager.go:414] "containerMap: removing container" podUID="1cf97eaa-077b-4652-8907-acd8bfbadade" containerName="maas-api" Nov 02 14:31:36.090643 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.090594 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf97eaa-077b-4652-8907-acd8bfbadade" containerName="maas-api" Nov 02 14:31:36.090643 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.090618 4866 cpu_manager.go:414] "containerMap: removing container" podUID="41e0f03e-11af-465d-aef2-8c3ef52a06bf" containerName="payload-processing" Nov 02 14:31:36.090643 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.090623 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e0f03e-11af-465d-aef2-8c3ef52a06bf" containerName="payload-processing" Nov 02 14:31:36.090739 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.090733 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e0f03e-11af-465d-aef2-8c3ef52a06bf" containerName="payload-processing" Nov 02 14:31:36.090782 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.090747 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf97eaa-077b-4652-8907-acd8bfbadade" containerName="maas-api" Nov 02 14:31:36.094127 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.094107 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" Nov 02 14:31:36.110808 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.110780 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-e07583-serving-cert\"" Nov 02 14:31:36.113198 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.113171 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk"] Nov 02 14:31:36.117941 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.117914 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2"] Nov 02 14:31:36.126568 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.126530 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d0c9b9-78dbf86775-q5dfk"] Nov 02 14:31:36.255041 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.254938 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41e0f03e-11af-465d-aef2-8c3ef52a06bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e0f03e-11af-465d-aef2-8c3ef52a06bf-plugins-config-volume\") pod \"41e0f03e-11af-465d-aef2-8c3ef52a06bf\" (UID: \"41e0f03e-11af-465d-aef2-8c3ef52a06bf\") " Nov 02 14:31:36.255202 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.255068 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e0f03e-11af-465d-aef2-8c3ef52a06bf-kube-api-access-vx7nx\" (UniqueName: \"kubernetes.io/projected/41e0f03e-11af-465d-aef2-8c3ef52a06bf-kube-api-access-vx7nx\") pod \"41e0f03e-11af-465d-aef2-8c3ef52a06bf\" (UID: \"41e0f03e-11af-465d-aef2-8c3ef52a06bf\") " Nov 02 14:31:36.255340 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.255325 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6803eccc-186b-4560-9dab-82bee489a940-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2\" (UID: \"6803eccc-186b-4560-9dab-82bee489a940\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" Nov 02 14:31:36.255392 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.255358 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6803eccc-186b-4560-9dab-82bee489a940-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2\" (UID: \"6803eccc-186b-4560-9dab-82bee489a940\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" Nov 02 14:31:36.255438 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.255409 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e0f03e-11af-465d-aef2-8c3ef52a06bf-plugins-config-volume" pod "41e0f03e-11af-465d-aef2-8c3ef52a06bf" (UID: "41e0f03e-11af-465d-aef2-8c3ef52a06bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:31:36.255515 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.255494 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6803eccc-186b-4560-9dab-82bee489a940-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2\" (UID: \"6803eccc-186b-4560-9dab-82bee489a940\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" Nov 02 14:31:36.255591 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.255574 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6lzl\" (UniqueName: \"kubernetes.io/projected/6803eccc-186b-4560-9dab-82bee489a940-kube-api-access-t6lzl\") pod \"maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2\" (UID: \"6803eccc-186b-4560-9dab-82bee489a940\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" Nov 02 14:31:36.255812 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.255794 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e0f03e-11af-465d-aef2-8c3ef52a06bf-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:31:36.257421 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.257398 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e0f03e-11af-465d-aef2-8c3ef52a06bf-kube-api-access-vx7nx" pod "41e0f03e-11af-465d-aef2-8c3ef52a06bf" (UID: "41e0f03e-11af-465d-aef2-8c3ef52a06bf"). InnerVolumeSpecName "kube-api-access-vx7nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:31:36.357020 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.356976 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6803eccc-186b-4560-9dab-82bee489a940-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2\" (UID: \"6803eccc-186b-4560-9dab-82bee489a940\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" Nov 02 14:31:36.357253 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.357039 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6803eccc-186b-4560-9dab-82bee489a940-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2\" (UID: \"6803eccc-186b-4560-9dab-82bee489a940\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" Nov 02 14:31:36.357253 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.357106 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6803eccc-186b-4560-9dab-82bee489a940-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2\" (UID: \"6803eccc-186b-4560-9dab-82bee489a940\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" Nov 02 14:31:36.357253 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.357135 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6lzl\" (UniqueName: \"kubernetes.io/projected/6803eccc-186b-4560-9dab-82bee489a940-kube-api-access-t6lzl\") pod \"maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2\" (UID: \"6803eccc-186b-4560-9dab-82bee489a940\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" Nov 02 14:31:36.357253 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:31:36.357221 4866 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-serving-cert: secret "maas-api-e2e-shared-a-w3-e07583-serving-cert" not found Nov 02 14:31:36.357445 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:31:36.357306 4866 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6803eccc-186b-4560-9dab-82bee489a940-maas-api-tls podName:6803eccc-186b-4560-9dab-82bee489a940 nodeName:}" failed. No retries permitted until 2026-11-02 14:31:36.857282407 +0000 UTC m=+2062.041129724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6803eccc-186b-4560-9dab-82bee489a940-maas-api-tls") pod "maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" (UID: "6803eccc-186b-4560-9dab-82bee489a940") : secret "maas-api-e2e-shared-a-w3-e07583-serving-cert" not found Nov 02 14:31:36.357445 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.357332 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx7nx\" (UniqueName: \"kubernetes.io/projected/41e0f03e-11af-465d-aef2-8c3ef52a06bf-kube-api-access-vx7nx\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:31:36.357829 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.357736 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6803eccc-186b-4560-9dab-82bee489a940-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2\" (UID: \"6803eccc-186b-4560-9dab-82bee489a940\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" Nov 02 14:31:36.359812 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.359789 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6803eccc-186b-4560-9dab-82bee489a940-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2\" (UID: \"6803eccc-186b-4560-9dab-82bee489a940\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" Nov 02 14:31:36.382542 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.382506 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6lzl\" (UniqueName: \"kubernetes.io/projected/6803eccc-186b-4560-9dab-82bee489a940-kube-api-access-t6lzl\") pod \"maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2\" (UID: \"6803eccc-186b-4560-9dab-82bee489a940\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" Nov 02 14:31:36.844082 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.844032 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx" podUID="41e0f03e-11af-465d-aef2-8c3ef52a06bf" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.115:50052\" within 1s: context deadline exceeded" Nov 02 14:31:36.862589 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.862562 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6803eccc-186b-4560-9dab-82bee489a940-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2\" (UID: \"6803eccc-186b-4560-9dab-82bee489a940\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" Nov 02 14:31:36.864981 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.864961 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6803eccc-186b-4560-9dab-82bee489a940-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2\" (UID: \"6803eccc-186b-4560-9dab-82bee489a940\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" Nov 02 14:31:36.949590 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:36.949550 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cf97eaa-077b-4652-8907-acd8bfbadade" path="/var/lib/kubelet/pods/1cf97eaa-077b-4652-8907-acd8bfbadade/volumes" Nov 02 14:31:37.008189 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:37.008144 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" Nov 02 14:31:37.064684 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:37.062091 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx" event={"ID":"41e0f03e-11af-465d-aef2-8c3ef52a06bf","Type":"ContainerDied","Data":"3356a772a4e9e8472b93130f0fc5f753db7b921483a949a8a5da0f08283196cf"} Nov 02 14:31:37.064684 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:37.062194 4866 scope.go:122] "RemoveContainer" containerID="df53f7a442a806c5aa847f8f7042db5fa3e1799fff160df74c1ab26cf9f70c87" Nov 02 14:31:37.065775 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:37.062197 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx" Nov 02 14:31:37.119563 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:37.119479 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx"] Nov 02 14:31:37.124205 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:37.124167 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d0c9b9-5fffc7bfd9-bdzlx"] Nov 02 14:31:37.329721 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:37.329684 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2"] Nov 02 14:31:37.332129 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:31:37.332098 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6803eccc_186b_4560_9dab_82bee489a940.slice/crio-fcb54ccd7c1cfd102bc13b50433f126adad7a16a07f884579349a02e7923ab7d WatchSource:0}: Error finding container fcb54ccd7c1cfd102bc13b50433f126adad7a16a07f884579349a02e7923ab7d: Status 404 returned error can't find the container with id fcb54ccd7c1cfd102bc13b50433f126adad7a16a07f884579349a02e7923ab7d Nov 02 14:31:37.333814 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:37.333793 4866 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:31:38.090357 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:38.090245 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" event={"ID":"6803eccc-186b-4560-9dab-82bee489a940","Type":"ContainerStarted","Data":"12848d78d5ca350f59f6e2a798d60a0e89d5b78ab9c999286d62b7441568e068"} Nov 02 14:31:38.090357 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:38.090311 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" event={"ID":"6803eccc-186b-4560-9dab-82bee489a940","Type":"ContainerStarted","Data":"fcb54ccd7c1cfd102bc13b50433f126adad7a16a07f884579349a02e7923ab7d"} Nov 02 14:31:38.090857 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:38.090448 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" Nov 02 14:31:38.134311 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:38.134255 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" podStartSLOduration=1.766746839 podStartE2EDuration="2.134242223s" totalImagesPullingTime="367.495384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:31:38.133104257 +0000 UTC m=+2063.316951606" watchObservedRunningTime="2026-11-02 14:31:38.134242223 +0000 UTC m=+2063.318089560" Nov 02 14:31:38.949753 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:38.949714 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41e0f03e-11af-465d-aef2-8c3ef52a06bf" path="/var/lib/kubelet/pods/41e0f03e-11af-465d-aef2-8c3ef52a06bf/volumes" Nov 02 14:31:44.103009 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:44.102978 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" Nov 02 14:31:46.417900 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:46.417850 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl"] Nov 02 14:31:46.448388 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:46.448353 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z"] Nov 02 14:31:51.418732 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:51.418680 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl" podUID="505c92f8-9033-4346-a88d-5bde108f70eb" containerName="payload-processing" containerID="cri-o://9f527872a72a84e74e806d83f1c6dc0f06626d15f505d10358a4d1b7064671a0" gracePeriod=25 Nov 02 14:31:51.449553 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:51.449508 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z" podUID="9bf6a036-4e2c-4343-bff1-21a55872af0a" containerName="payload-pre-processing" containerID="cri-o://80457be6e10d1e4f3edd2bb788fd3f78fbd8fabdeafd50d0691c709e76be9ba2" gracePeriod=25 Nov 02 14:31:51.766564 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:51.766535 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl" Nov 02 14:31:51.770080 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:51.770057 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z" Nov 02 14:31:51.851780 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:51.851738 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf6a036-4e2c-4343-bff1-21a55872af0a-kube-api-access-2n6qk\" (UniqueName: \"kubernetes.io/projected/9bf6a036-4e2c-4343-bff1-21a55872af0a-kube-api-access-2n6qk\") pod \"9bf6a036-4e2c-4343-bff1-21a55872af0a\" (UID: \"9bf6a036-4e2c-4343-bff1-21a55872af0a\") " Nov 02 14:31:51.858106 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:51.857607 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bf6a036-4e2c-4343-bff1-21a55872af0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf6a036-4e2c-4343-bff1-21a55872af0a-plugins-config-volume\") pod \"9bf6a036-4e2c-4343-bff1-21a55872af0a\" (UID: \"9bf6a036-4e2c-4343-bff1-21a55872af0a\") " Nov 02 14:31:51.858417 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:51.858082 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf6a036-4e2c-4343-bff1-21a55872af0a-plugins-config-volume" pod "9bf6a036-4e2c-4343-bff1-21a55872af0a" (UID: "9bf6a036-4e2c-4343-bff1-21a55872af0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:31:51.858627 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:51.858606 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/505c92f8-9033-4346-a88d-5bde108f70eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/505c92f8-9033-4346-a88d-5bde108f70eb-plugins-config-volume\") pod \"505c92f8-9033-4346-a88d-5bde108f70eb\" (UID: \"505c92f8-9033-4346-a88d-5bde108f70eb\") " Nov 02 14:31:51.858762 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:51.858749 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/505c92f8-9033-4346-a88d-5bde108f70eb-kube-api-access-9twnv\" (UniqueName: \"kubernetes.io/projected/505c92f8-9033-4346-a88d-5bde108f70eb-kube-api-access-9twnv\") pod \"505c92f8-9033-4346-a88d-5bde108f70eb\" (UID: \"505c92f8-9033-4346-a88d-5bde108f70eb\") " Nov 02 14:31:51.859089 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:51.859041 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/505c92f8-9033-4346-a88d-5bde108f70eb-plugins-config-volume" pod "505c92f8-9033-4346-a88d-5bde108f70eb" (UID: "505c92f8-9033-4346-a88d-5bde108f70eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:31:51.859659 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:51.859481 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf6a036-4e2c-4343-bff1-21a55872af0a-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:31:51.859659 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:51.859508 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/505c92f8-9033-4346-a88d-5bde108f70eb-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:31:51.889791 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:51.889724 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505c92f8-9033-4346-a88d-5bde108f70eb-kube-api-access-9twnv" pod "505c92f8-9033-4346-a88d-5bde108f70eb" (UID: "505c92f8-9033-4346-a88d-5bde108f70eb"). InnerVolumeSpecName "kube-api-access-9twnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:31:51.900332 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:51.892126 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf6a036-4e2c-4343-bff1-21a55872af0a-kube-api-access-2n6qk" pod "9bf6a036-4e2c-4343-bff1-21a55872af0a" (UID: "9bf6a036-4e2c-4343-bff1-21a55872af0a"). InnerVolumeSpecName "kube-api-access-2n6qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:31:51.961159 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:51.961116 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9twnv\" (UniqueName: \"kubernetes.io/projected/505c92f8-9033-4346-a88d-5bde108f70eb-kube-api-access-9twnv\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:31:51.961159 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:51.961159 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n6qk\" (UniqueName: \"kubernetes.io/projected/9bf6a036-4e2c-4343-bff1-21a55872af0a-kube-api-access-2n6qk\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:31:52.178478 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.178392 4866 generic.go:356] "Generic (PLEG): container finished" podID="9bf6a036-4e2c-4343-bff1-21a55872af0a" containerID="80457be6e10d1e4f3edd2bb788fd3f78fbd8fabdeafd50d0691c709e76be9ba2" exitCode=0 Nov 02 14:31:52.178478 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.178442 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z" Nov 02 14:31:52.178684 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.178474 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z" event={"ID":"9bf6a036-4e2c-4343-bff1-21a55872af0a","Type":"ContainerDied","Data":"80457be6e10d1e4f3edd2bb788fd3f78fbd8fabdeafd50d0691c709e76be9ba2"} Nov 02 14:31:52.178684 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.178535 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z" event={"ID":"9bf6a036-4e2c-4343-bff1-21a55872af0a","Type":"ContainerDied","Data":"96ea62b1eb9a3439bbe96900a20009bb1299eacf20dac309bb86715755dbf37b"} Nov 02 14:31:52.178684 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.178572 4866 scope.go:122] "RemoveContainer" containerID="80457be6e10d1e4f3edd2bb788fd3f78fbd8fabdeafd50d0691c709e76be9ba2" Nov 02 14:31:52.179911 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.179885 4866 generic.go:356] "Generic (PLEG): container finished" podID="505c92f8-9033-4346-a88d-5bde108f70eb" containerID="9f527872a72a84e74e806d83f1c6dc0f06626d15f505d10358a4d1b7064671a0" exitCode=0 Nov 02 14:31:52.180062 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.179917 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl" event={"ID":"505c92f8-9033-4346-a88d-5bde108f70eb","Type":"ContainerDied","Data":"9f527872a72a84e74e806d83f1c6dc0f06626d15f505d10358a4d1b7064671a0"} Nov 02 14:31:52.180062 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.179938 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl" Nov 02 14:31:52.180062 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.179989 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl" event={"ID":"505c92f8-9033-4346-a88d-5bde108f70eb","Type":"ContainerDied","Data":"7423781e535ae481a01e2cbf0fc5ed28112e6fa521542632f2a7e7bfe8eff20c"} Nov 02 14:31:52.188663 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.188639 4866 scope.go:122] "RemoveContainer" containerID="80457be6e10d1e4f3edd2bb788fd3f78fbd8fabdeafd50d0691c709e76be9ba2" Nov 02 14:31:52.189097 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:31:52.189072 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80457be6e10d1e4f3edd2bb788fd3f78fbd8fabdeafd50d0691c709e76be9ba2\": container with ID starting with 80457be6e10d1e4f3edd2bb788fd3f78fbd8fabdeafd50d0691c709e76be9ba2 not found: ID does not exist" containerID="80457be6e10d1e4f3edd2bb788fd3f78fbd8fabdeafd50d0691c709e76be9ba2" Nov 02 14:31:52.189236 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.189105 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80457be6e10d1e4f3edd2bb788fd3f78fbd8fabdeafd50d0691c709e76be9ba2"} err="failed to get container status \"80457be6e10d1e4f3edd2bb788fd3f78fbd8fabdeafd50d0691c709e76be9ba2\": rpc error: code = NotFound desc = could not find container \"80457be6e10d1e4f3edd2bb788fd3f78fbd8fabdeafd50d0691c709e76be9ba2\": container with ID starting with 80457be6e10d1e4f3edd2bb788fd3f78fbd8fabdeafd50d0691c709e76be9ba2 not found: ID does not exist" Nov 02 14:31:52.189236 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.189124 4866 scope.go:122] "RemoveContainer" containerID="9f527872a72a84e74e806d83f1c6dc0f06626d15f505d10358a4d1b7064671a0" Nov 02 14:31:52.199518 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.199498 4866 scope.go:122] "RemoveContainer" containerID="9f527872a72a84e74e806d83f1c6dc0f06626d15f505d10358a4d1b7064671a0" Nov 02 14:31:52.199843 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:31:52.199820 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f527872a72a84e74e806d83f1c6dc0f06626d15f505d10358a4d1b7064671a0\": container with ID starting with 9f527872a72a84e74e806d83f1c6dc0f06626d15f505d10358a4d1b7064671a0 not found: ID does not exist" containerID="9f527872a72a84e74e806d83f1c6dc0f06626d15f505d10358a4d1b7064671a0" Nov 02 14:31:52.199892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.199853 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f527872a72a84e74e806d83f1c6dc0f06626d15f505d10358a4d1b7064671a0"} err="failed to get container status \"9f527872a72a84e74e806d83f1c6dc0f06626d15f505d10358a4d1b7064671a0\": rpc error: code = NotFound desc = could not find container \"9f527872a72a84e74e806d83f1c6dc0f06626d15f505d10358a4d1b7064671a0\": container with ID starting with 9f527872a72a84e74e806d83f1c6dc0f06626d15f505d10358a4d1b7064671a0 not found: ID does not exist" Nov 02 14:31:52.230619 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.230582 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl"] Nov 02 14:31:52.237854 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.237821 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-928a7a-79d54fb94f-7dlcl"] Nov 02 14:31:52.274687 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.274650 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z"] Nov 02 14:31:52.278237 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.278210 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-928a7a-784d585f5-cg49z"] Nov 02 14:31:52.947419 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.947387 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="505c92f8-9033-4346-a88d-5bde108f70eb" path="/var/lib/kubelet/pods/505c92f8-9033-4346-a88d-5bde108f70eb/volumes" Nov 02 14:31:52.947848 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:31:52.947810 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf6a036-4e2c-4343-bff1-21a55872af0a" path="/var/lib/kubelet/pods/9bf6a036-4e2c-4343-bff1-21a55872af0a/volumes" Nov 02 14:32:06.010563 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:32:06.010526 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:32:06.859108 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:06.859073 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6"] Nov 02 14:32:06.859664 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:06.859651 4866 cpu_manager.go:414] "containerMap: removing container" podUID="505c92f8-9033-4346-a88d-5bde108f70eb" containerName="payload-processing" Nov 02 14:32:06.859707 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:06.859665 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="505c92f8-9033-4346-a88d-5bde108f70eb" containerName="payload-processing" Nov 02 14:32:06.859745 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:06.859714 4866 cpu_manager.go:414] "containerMap: removing container" podUID="9bf6a036-4e2c-4343-bff1-21a55872af0a" containerName="payload-pre-processing" Nov 02 14:32:06.859745 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:06.859719 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf6a036-4e2c-4343-bff1-21a55872af0a" containerName="payload-pre-processing" Nov 02 14:32:06.859808 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:06.859797 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf6a036-4e2c-4343-bff1-21a55872af0a" containerName="payload-pre-processing" Nov 02 14:32:06.859840 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:06.859810 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="505c92f8-9033-4346-a88d-5bde108f70eb" containerName="payload-processing" Nov 02 14:32:06.864681 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:06.864655 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:06.892599 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:06.892565 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-623b4b\"/\"openshift-service-ca.crt\"" Nov 02 14:32:06.892753 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:06.892676 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-623b4b\"/\"kube-root-ca.crt\"" Nov 02 14:32:06.892753 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:06.892717 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-623b4b\"/\"default-dockercfg-79j45\"" Nov 02 14:32:06.892753 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:06.892723 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-623b4b\"/\"rate-test-model-623b4b-kserve-self-signed-certs\"" Nov 02 14:32:06.897345 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:06.897317 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6"] Nov 02 14:32:06.957038 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:06.957004 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-home\") pod \"rate-test-model-623b4b-kserve-c444f69d5-qmrh6\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:06.957248 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:06.957043 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-dshm\") pod \"rate-test-model-623b4b-kserve-c444f69d5-qmrh6\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:06.957319 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:06.957296 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-tls-certs\") pod \"rate-test-model-623b4b-kserve-c444f69d5-qmrh6\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:06.957380 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:06.957356 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-tmp-dir\") pod \"rate-test-model-623b4b-kserve-c444f69d5-qmrh6\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:06.957426 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:06.957408 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-model-cache\") pod \"rate-test-model-623b4b-kserve-c444f69d5-qmrh6\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:06.957487 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:06.957437 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xndg\" (UniqueName: \"kubernetes.io/projected/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-kube-api-access-4xndg\") pod \"rate-test-model-623b4b-kserve-c444f69d5-qmrh6\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:07.058811 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:07.058772 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-tls-certs\") pod \"rate-test-model-623b4b-kserve-c444f69d5-qmrh6\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:07.059329 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:07.058833 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-tmp-dir\") pod \"rate-test-model-623b4b-kserve-c444f69d5-qmrh6\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:07.059329 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:07.058863 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-model-cache\") pod \"rate-test-model-623b4b-kserve-c444f69d5-qmrh6\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:07.059329 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:07.058888 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xndg\" (UniqueName: \"kubernetes.io/projected/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-kube-api-access-4xndg\") pod \"rate-test-model-623b4b-kserve-c444f69d5-qmrh6\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:07.059329 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:07.059016 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-home\") pod \"rate-test-model-623b4b-kserve-c444f69d5-qmrh6\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:07.059329 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:07.059228 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-dshm\") pod \"rate-test-model-623b4b-kserve-c444f69d5-qmrh6\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:07.059329 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:07.059295 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-tmp-dir\") pod \"rate-test-model-623b4b-kserve-c444f69d5-qmrh6\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:07.059329 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:07.059320 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-model-cache\") pod \"rate-test-model-623b4b-kserve-c444f69d5-qmrh6\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:07.059708 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:07.059385 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-home\") pod \"rate-test-model-623b4b-kserve-c444f69d5-qmrh6\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:07.061396 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:07.061372 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-dshm\") pod \"rate-test-model-623b4b-kserve-c444f69d5-qmrh6\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:07.061627 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:07.061606 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-tls-certs\") pod \"rate-test-model-623b4b-kserve-c444f69d5-qmrh6\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:07.082442 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:07.082409 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xndg\" (UniqueName: \"kubernetes.io/projected/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-kube-api-access-4xndg\") pod \"rate-test-model-623b4b-kserve-c444f69d5-qmrh6\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:07.183282 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:07.183246 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:07.522744 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:07.522714 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6"] Nov 02 14:32:07.524218 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:32:07.524188 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91e00078_7b8d_41d9_b385_4ee8a4fe3b50.slice/crio-69a919f8fcfb523ad4b8d619bdcbe3e1cf8c7ed3466324da07f871cc1e194cce WatchSource:0}: Error finding container 69a919f8fcfb523ad4b8d619bdcbe3e1cf8c7ed3466324da07f871cc1e194cce: Status 404 returned error can't find the container with id 69a919f8fcfb523ad4b8d619bdcbe3e1cf8c7ed3466324da07f871cc1e194cce Nov 02 14:32:08.290430 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:08.290396 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" event={"ID":"91e00078-7b8d-41d9-b385-4ee8a4fe3b50","Type":"ContainerStarted","Data":"3c224ede1af5ed3379e935ae833083eecdc91d753e72cefa50fe75455a134ff2"} Nov 02 14:32:08.290842 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:08.290441 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" event={"ID":"91e00078-7b8d-41d9-b385-4ee8a4fe3b50","Type":"ContainerStarted","Data":"69a919f8fcfb523ad4b8d619bdcbe3e1cf8c7ed3466324da07f871cc1e194cce"} Nov 02 14:32:08.332686 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:08.332613 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" podStartSLOduration=2.332595816 podStartE2EDuration="2.332595816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:32:08.331430188 +0000 UTC m=+2093.515277524" watchObservedRunningTime="2026-11-02 14:32:08.332595816 +0000 UTC m=+2093.516443156" Nov 02 14:32:09.184193 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:09.184163 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:12.901687 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:12.901603 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l"] Nov 02 14:32:12.905967 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:12.905930 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:12.921019 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:12.920991 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-5ba53f-openshift-default-dockercfg-dzllq\"" Nov 02 14:32:12.936402 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:12.936372 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l"] Nov 02 14:32:12.955962 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:12.955908 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-token\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:12.956137 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:12.956085 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-envoy\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:12.956137 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:12.956131 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-data\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:12.956305 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:12.956288 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-podinfo\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:12.956438 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:12.956424 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-workload-certs\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:12.956495 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:12.956459 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc409ca-3338-4474-bad2-aeb03d23aedc-istiod-ca-cert\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:12.956552 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:12.956491 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-workload-socket\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:12.956552 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:12.956518 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncpjr\" (UniqueName: \"kubernetes.io/projected/6bc409ca-3338-4474-bad2-aeb03d23aedc-kube-api-access-ncpjr\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:12.956552 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:12.956546 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-credential-socket\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.057787 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.057748 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-envoy\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.057978 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.057869 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-data\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.058049 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.057994 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-podinfo\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.058108 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.058093 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-workload-certs\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.058164 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.058147 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc409ca-3338-4474-bad2-aeb03d23aedc-istiod-ca-cert\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.058205 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.058187 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-workload-socket\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.058267 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.058215 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncpjr\" (UniqueName: \"kubernetes.io/projected/6bc409ca-3338-4474-bad2-aeb03d23aedc-kube-api-access-ncpjr\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.058267 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.058246 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-credential-socket\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.058373 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.058322 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-token\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.058451 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.058323 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-data\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.058544 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.058514 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-workload-certs\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.058899 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.058868 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-credential-socket\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.059064 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.058897 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc409ca-3338-4474-bad2-aeb03d23aedc-istiod-ca-cert\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.059064 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.058877 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-workload-socket\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.060453 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.060427 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-envoy\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.060614 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.060599 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-podinfo\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.082057 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.082028 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncpjr\" (UniqueName: \"kubernetes.io/projected/6bc409ca-3338-4474-bad2-aeb03d23aedc-kube-api-access-ncpjr\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.082211 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.082082 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-token\") pod \"e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.225032 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.224998 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:13.545256 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.545226 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l"] Nov 02 14:32:13.546468 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:32:13.546443 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc409ca_3338_4474_bad2_aeb03d23aedc.slice/crio-0899899f4117ca3f9a1401023199ab9909d1a49e45cd623e942c1ab58a7c176f WatchSource:0}: Error finding container 0899899f4117ca3f9a1401023199ab9909d1a49e45cd623e942c1ab58a7c176f: Status 404 returned error can't find the container with id 0899899f4117ca3f9a1401023199ab9909d1a49e45cd623e942c1ab58a7c176f Nov 02 14:32:13.549298 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.549267 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:32:13.549361 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.549339 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:32:13.549422 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:13.549370 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:32:14.330467 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:14.330420 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" event={"ID":"6bc409ca-3338-4474-bad2-aeb03d23aedc","Type":"ContainerStarted","Data":"17c24af6f40d321f5fc7021d3fab583d05511ffcc6a4af356fb15f4159f89cd0"} Nov 02 14:32:14.330874 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:14.330491 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" event={"ID":"6bc409ca-3338-4474-bad2-aeb03d23aedc","Type":"ContainerStarted","Data":"0899899f4117ca3f9a1401023199ab9909d1a49e45cd623e942c1ab58a7c176f"} Nov 02 14:32:14.389715 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:14.389659 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" podStartSLOduration=2.389645079 podStartE2EDuration="2.389645079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:32:14.386459831 +0000 UTC m=+2099.570307168" watchObservedRunningTime="2026-11-02 14:32:14.389645079 +0000 UTC m=+2099.573492416" Nov 02 14:32:15.225347 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:15.225306 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:15.231466 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:15.231435 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:15.337005 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:15.336977 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:15.338022 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:15.338000 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:32:15.509782 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:15.509700 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:32:15.525466 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:15.525437 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:32:17.184285 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:17.184232 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:17.198618 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:17.198591 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:17.360134 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:17.360107 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:32:18.891989 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:18.891956 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z"] Nov 02 14:32:18.901391 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:18.899841 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" Nov 02 14:32:18.918492 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:18.918460 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z"] Nov 02 14:32:18.926206 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:18.926172 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-5ba53f\"" Nov 02 14:32:18.926393 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:18.926314 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-5ba53f-dockercfg-9czwx\"" Nov 02 14:32:19.038522 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:19.038477 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9rpz\" (UniqueName: \"kubernetes.io/projected/dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8-kube-api-access-j9rpz\") pod \"payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z\" (UID: \"dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" Nov 02 14:32:19.038743 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:19.038714 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z\" (UID: \"dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" Nov 02 14:32:19.139743 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:19.139686 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z\" (UID: \"dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" Nov 02 14:32:19.139990 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:19.139904 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9rpz\" (UniqueName: \"kubernetes.io/projected/dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8-kube-api-access-j9rpz\") pod \"payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z\" (UID: \"dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" Nov 02 14:32:19.141375 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:19.141343 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z\" (UID: \"dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" Nov 02 14:32:19.163481 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:19.163398 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9rpz\" (UniqueName: \"kubernetes.io/projected/dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8-kube-api-access-j9rpz\") pod \"payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z\" (UID: \"dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" Nov 02 14:32:19.216118 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:19.216088 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" Nov 02 14:32:19.526400 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:19.526372 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z"] Nov 02 14:32:19.528238 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:32:19.528203 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2bd27f_ce10_41d9_9c35_ff8b6995c6a8.slice/crio-1c45a53c0b5086694f570c8b8121c14e7c7983ac2f9e6ecc72c72835cc2cceff WatchSource:0}: Error finding container 1c45a53c0b5086694f570c8b8121c14e7c7983ac2f9e6ecc72c72835cc2cceff: Status 404 returned error can't find the container with id 1c45a53c0b5086694f570c8b8121c14e7c7983ac2f9e6ecc72c72835cc2cceff Nov 02 14:32:20.369387 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:20.369350 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" event={"ID":"dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8","Type":"ContainerStarted","Data":"4fe080d0bbcd7faf854b30b02e62087a5ac40cd696b4e2c49646f10a08cdcdf3"} Nov 02 14:32:20.369831 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:20.369402 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" event={"ID":"dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8","Type":"ContainerStarted","Data":"1c45a53c0b5086694f570c8b8121c14e7c7983ac2f9e6ecc72c72835cc2cceff"} Nov 02 14:32:20.369831 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:20.369434 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" Nov 02 14:32:20.415910 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:20.415852 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" podStartSLOduration=2.41583696 podStartE2EDuration="2.41583696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:32:20.413335437 +0000 UTC m=+2105.597182776" watchObservedRunningTime="2026-11-02 14:32:20.41583696 +0000 UTC m=+2105.599684297" Nov 02 14:32:22.454856 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:22.454818 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242"] Nov 02 14:32:22.455725 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:22.455419 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" podUID="e62231f6-df12-4423-9b82-04db2235d365" containerName="maas-api" containerID="cri-o://81f9ea67633e8c3c8b2c782d874fb3d32e6f44d83659a515df30a83e81d1d294" gracePeriod=30 Nov 02 14:32:22.742063 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:22.732979 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" Nov 02 14:32:22.889452 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:22.889394 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e62231f6-df12-4423-9b82-04db2235d365-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62231f6-df12-4423-9b82-04db2235d365-metrics-certs\") pod \"e62231f6-df12-4423-9b82-04db2235d365\" (UID: \"e62231f6-df12-4423-9b82-04db2235d365\") " Nov 02 14:32:22.889650 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:22.889494 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e62231f6-df12-4423-9b82-04db2235d365-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62231f6-df12-4423-9b82-04db2235d365-maas-api-tls\") pod \"e62231f6-df12-4423-9b82-04db2235d365\" (UID: \"e62231f6-df12-4423-9b82-04db2235d365\") " Nov 02 14:32:22.889650 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:22.889601 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62231f6-df12-4423-9b82-04db2235d365-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62231f6-df12-4423-9b82-04db2235d365-openshift-service-ca\") pod \"e62231f6-df12-4423-9b82-04db2235d365\" (UID: \"e62231f6-df12-4423-9b82-04db2235d365\") " Nov 02 14:32:22.889760 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:22.889649 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62231f6-df12-4423-9b82-04db2235d365-kube-api-access-plrf6\" (UniqueName: \"kubernetes.io/projected/e62231f6-df12-4423-9b82-04db2235d365-kube-api-access-plrf6\") pod \"e62231f6-df12-4423-9b82-04db2235d365\" (UID: \"e62231f6-df12-4423-9b82-04db2235d365\") " Nov 02 14:32:22.890109 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:22.890035 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62231f6-df12-4423-9b82-04db2235d365-openshift-service-ca" pod "e62231f6-df12-4423-9b82-04db2235d365" (UID: "e62231f6-df12-4423-9b82-04db2235d365"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:32:22.890581 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:22.890545 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62231f6-df12-4423-9b82-04db2235d365-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:32:22.905418 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:22.895499 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62231f6-df12-4423-9b82-04db2235d365-metrics-certs" pod "e62231f6-df12-4423-9b82-04db2235d365" (UID: "e62231f6-df12-4423-9b82-04db2235d365"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:32:22.910069 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:22.907971 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62231f6-df12-4423-9b82-04db2235d365-kube-api-access-plrf6" pod "e62231f6-df12-4423-9b82-04db2235d365" (UID: "e62231f6-df12-4423-9b82-04db2235d365"). InnerVolumeSpecName "kube-api-access-plrf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:32:22.910069 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:22.908090 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62231f6-df12-4423-9b82-04db2235d365-maas-api-tls" pod "e62231f6-df12-4423-9b82-04db2235d365" (UID: "e62231f6-df12-4423-9b82-04db2235d365"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:32:22.991901 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:22.991867 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plrf6\" (UniqueName: \"kubernetes.io/projected/e62231f6-df12-4423-9b82-04db2235d365-kube-api-access-plrf6\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:32:22.991901 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:22.991899 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62231f6-df12-4423-9b82-04db2235d365-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:32:22.992134 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:22.991912 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62231f6-df12-4423-9b82-04db2235d365-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:32:23.388229 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:23.388125 4866 generic.go:356] "Generic (PLEG): container finished" podID="e62231f6-df12-4423-9b82-04db2235d365" containerID="81f9ea67633e8c3c8b2c782d874fb3d32e6f44d83659a515df30a83e81d1d294" exitCode=0 Nov 02 14:32:23.388229 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:23.388177 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" Nov 02 14:32:23.388229 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:23.388212 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" event={"ID":"e62231f6-df12-4423-9b82-04db2235d365","Type":"ContainerDied","Data":"81f9ea67633e8c3c8b2c782d874fb3d32e6f44d83659a515df30a83e81d1d294"} Nov 02 14:32:23.388512 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:23.388266 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242" event={"ID":"e62231f6-df12-4423-9b82-04db2235d365","Type":"ContainerDied","Data":"3d9ac64c9b56035102d54ef74e75b7498fc1eb77de2122efb8ed1dd417e881f7"} Nov 02 14:32:23.388512 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:23.388290 4866 scope.go:122] "RemoveContainer" containerID="81f9ea67633e8c3c8b2c782d874fb3d32e6f44d83659a515df30a83e81d1d294" Nov 02 14:32:23.398268 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:23.398249 4866 scope.go:122] "RemoveContainer" containerID="81f9ea67633e8c3c8b2c782d874fb3d32e6f44d83659a515df30a83e81d1d294" Nov 02 14:32:23.398576 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:32:23.398554 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f9ea67633e8c3c8b2c782d874fb3d32e6f44d83659a515df30a83e81d1d294\": container with ID starting with 81f9ea67633e8c3c8b2c782d874fb3d32e6f44d83659a515df30a83e81d1d294 not found: ID does not exist" containerID="81f9ea67633e8c3c8b2c782d874fb3d32e6f44d83659a515df30a83e81d1d294" Nov 02 14:32:23.398626 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:23.398583 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f9ea67633e8c3c8b2c782d874fb3d32e6f44d83659a515df30a83e81d1d294"} err="failed to get container status \"81f9ea67633e8c3c8b2c782d874fb3d32e6f44d83659a515df30a83e81d1d294\": rpc error: code = NotFound desc = could not find container \"81f9ea67633e8c3c8b2c782d874fb3d32e6f44d83659a515df30a83e81d1d294\": container with ID starting with 81f9ea67633e8c3c8b2c782d874fb3d32e6f44d83659a515df30a83e81d1d294 not found: ID does not exist" Nov 02 14:32:23.440781 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:23.440743 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242"] Nov 02 14:32:23.453682 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:23.453652 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-928a7a-7f49bcb5bf-rp242"] Nov 02 14:32:24.947724 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:24.947685 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62231f6-df12-4423-9b82-04db2235d365" path="/var/lib/kubelet/pods/e62231f6-df12-4423-9b82-04db2235d365/volumes" Nov 02 14:32:31.378838 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:31.378756 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" Nov 02 14:32:36.010371 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:32:36.010338 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:32:37.371123 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.371082 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg"] Nov 02 14:32:37.371959 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.371920 4866 cpu_manager.go:414] "containerMap: removing container" podUID="e62231f6-df12-4423-9b82-04db2235d365" containerName="maas-api" Nov 02 14:32:37.372073 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.371960 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62231f6-df12-4423-9b82-04db2235d365" containerName="maas-api" Nov 02 14:32:37.372175 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.372160 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62231f6-df12-4423-9b82-04db2235d365" containerName="maas-api" Nov 02 14:32:37.376794 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.376772 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" Nov 02 14:32:37.422046 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.422011 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"kube-root-ca.crt\"" Nov 02 14:32:37.426009 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.425977 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg"] Nov 02 14:32:37.426207 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.426189 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"routing-overlay\"" Nov 02 14:32:37.426413 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.426396 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9ba664\"" Nov 02 14:32:37.433372 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.433342 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"openshift-service-ca.crt\"" Nov 02 14:32:37.433372 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.433352 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"agc-cred-1ca87d03f382\"" Nov 02 14:32:37.433814 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.433790 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"payload-processing-external-model-e2e--d8f5123f-dockercfg-nl82m\"" Nov 02 14:32:37.488507 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.488472 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4991d22f-27fd-44d9-bcfe-2f991088c0bc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg\" (UID: \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" Nov 02 14:32:37.488507 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.488513 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4991d22f-27fd-44d9-bcfe-2f991088c0bc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg\" (UID: \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" Nov 02 14:32:37.488745 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.488632 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4991d22f-27fd-44d9-bcfe-2f991088c0bc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg\" (UID: \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" Nov 02 14:32:37.488745 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.488662 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stch8\" (UniqueName: \"kubernetes.io/projected/4991d22f-27fd-44d9-bcfe-2f991088c0bc-kube-api-access-stch8\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg\" (UID: \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" Nov 02 14:32:37.591473 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.590972 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4991d22f-27fd-44d9-bcfe-2f991088c0bc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg\" (UID: \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" Nov 02 14:32:37.591473 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.591031 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4991d22f-27fd-44d9-bcfe-2f991088c0bc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg\" (UID: \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" Nov 02 14:32:37.591473 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.591107 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4991d22f-27fd-44d9-bcfe-2f991088c0bc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg\" (UID: \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" Nov 02 14:32:37.591473 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.591133 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stch8\" (UniqueName: \"kubernetes.io/projected/4991d22f-27fd-44d9-bcfe-2f991088c0bc-kube-api-access-stch8\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg\" (UID: \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" Nov 02 14:32:37.614972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.608414 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4991d22f-27fd-44d9-bcfe-2f991088c0bc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg\" (UID: \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" Nov 02 14:32:37.614972 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.608961 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4991d22f-27fd-44d9-bcfe-2f991088c0bc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg\" (UID: \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" Nov 02 14:32:37.624188 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.623825 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4991d22f-27fd-44d9-bcfe-2f991088c0bc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg\" (UID: \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" Nov 02 14:32:37.631471 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.629998 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stch8\" (UniqueName: \"kubernetes.io/projected/4991d22f-27fd-44d9-bcfe-2f991088c0bc-kube-api-access-stch8\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg\" (UID: \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" Nov 02 14:32:37.691423 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:37.691386 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" Nov 02 14:32:38.000099 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:32:38.000072 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4991d22f_27fd_44d9_bcfe_2f991088c0bc.slice/crio-bb0b3c195a7b22e59b9140067ebf53aea188d1ddd1b4ab588f6d447aa207642f WatchSource:0}: Error finding container bb0b3c195a7b22e59b9140067ebf53aea188d1ddd1b4ab588f6d447aa207642f: Status 404 returned error can't find the container with id bb0b3c195a7b22e59b9140067ebf53aea188d1ddd1b4ab588f6d447aa207642f Nov 02 14:32:38.001028 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.000998 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg"] Nov 02 14:32:38.343610 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.343534 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq"] Nov 02 14:32:38.347746 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.347729 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq" Nov 02 14:32:38.366699 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.366668 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-90a96161\"" Nov 02 14:32:38.367453 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.367432 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-90a96161-dockercfg-hhfwx\"" Nov 02 14:32:38.371443 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.371417 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq"] Nov 02 14:32:38.415091 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.415043 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae683718-049c-4ea8-82da-4f97e31175ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq\" (UID: \"ae683718-049c-4ea8-82da-4f97e31175ac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq" Nov 02 14:32:38.415417 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.415232 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz9zl\" (UniqueName: \"kubernetes.io/projected/ae683718-049c-4ea8-82da-4f97e31175ac-kube-api-access-dz9zl\") pod \"payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq\" (UID: \"ae683718-049c-4ea8-82da-4f97e31175ac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq" Nov 02 14:32:38.469969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.469180 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m"] Nov 02 14:32:38.484966 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.475853 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m" Nov 02 14:32:38.499968 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.494265 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m"] Nov 02 14:32:38.520830 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.516650 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz9zl\" (UniqueName: \"kubernetes.io/projected/ae683718-049c-4ea8-82da-4f97e31175ac-kube-api-access-dz9zl\") pod \"payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq\" (UID: \"ae683718-049c-4ea8-82da-4f97e31175ac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq" Nov 02 14:32:38.520830 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.516874 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae683718-049c-4ea8-82da-4f97e31175ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq\" (UID: \"ae683718-049c-4ea8-82da-4f97e31175ac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq" Nov 02 14:32:38.521807 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.521776 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae683718-049c-4ea8-82da-4f97e31175ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq\" (UID: \"ae683718-049c-4ea8-82da-4f97e31175ac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq" Nov 02 14:32:38.525459 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.525423 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" event={"ID":"4991d22f-27fd-44d9-bcfe-2f991088c0bc","Type":"ContainerStarted","Data":"fdc96af376b40c3e4eabe3d95ddc9e2d0ad09f114cd54e637e12f874f8c8fdd2"} Nov 02 14:32:38.525592 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.525479 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" event={"ID":"4991d22f-27fd-44d9-bcfe-2f991088c0bc","Type":"ContainerStarted","Data":"bb0b3c195a7b22e59b9140067ebf53aea188d1ddd1b4ab588f6d447aa207642f"} Nov 02 14:32:38.525708 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.525693 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" Nov 02 14:32:38.544364 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.544331 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz9zl\" (UniqueName: \"kubernetes.io/projected/ae683718-049c-4ea8-82da-4f97e31175ac-kube-api-access-dz9zl\") pod \"payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq\" (UID: \"ae683718-049c-4ea8-82da-4f97e31175ac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq" Nov 02 14:32:38.574296 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.574236 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" podStartSLOduration=1.57422028 podStartE2EDuration="1.57422028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:32:38.571560574 +0000 UTC m=+2123.755407935" watchObservedRunningTime="2026-11-02 14:32:38.57422028 +0000 UTC m=+2123.758067616" Nov 02 14:32:38.618019 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.617913 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkd4r\" (UniqueName: \"kubernetes.io/projected/d73bd157-9369-406e-b2bf-d4cfa38a1d33-kube-api-access-wkd4r\") pod \"payload-processing-e2e-mt-90a96161-694f45b47c-t985m\" (UID: \"d73bd157-9369-406e-b2bf-d4cfa38a1d33\") " pod="openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m" Nov 02 14:32:38.618165 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.618082 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73bd157-9369-406e-b2bf-d4cfa38a1d33-plugins-config-volume\") pod \"payload-processing-e2e-mt-90a96161-694f45b47c-t985m\" (UID: \"d73bd157-9369-406e-b2bf-d4cfa38a1d33\") " pod="openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m" Nov 02 14:32:38.661167 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.661132 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq" Nov 02 14:32:38.719599 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.719558 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkd4r\" (UniqueName: \"kubernetes.io/projected/d73bd157-9369-406e-b2bf-d4cfa38a1d33-kube-api-access-wkd4r\") pod \"payload-processing-e2e-mt-90a96161-694f45b47c-t985m\" (UID: \"d73bd157-9369-406e-b2bf-d4cfa38a1d33\") " pod="openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m" Nov 02 14:32:38.719809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.719660 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73bd157-9369-406e-b2bf-d4cfa38a1d33-plugins-config-volume\") pod \"payload-processing-e2e-mt-90a96161-694f45b47c-t985m\" (UID: \"d73bd157-9369-406e-b2bf-d4cfa38a1d33\") " pod="openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m" Nov 02 14:32:38.720529 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.720502 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73bd157-9369-406e-b2bf-d4cfa38a1d33-plugins-config-volume\") pod \"payload-processing-e2e-mt-90a96161-694f45b47c-t985m\" (UID: \"d73bd157-9369-406e-b2bf-d4cfa38a1d33\") " pod="openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m" Nov 02 14:32:38.747926 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.747852 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkd4r\" (UniqueName: \"kubernetes.io/projected/d73bd157-9369-406e-b2bf-d4cfa38a1d33-kube-api-access-wkd4r\") pod \"payload-processing-e2e-mt-90a96161-694f45b47c-t985m\" (UID: \"d73bd157-9369-406e-b2bf-d4cfa38a1d33\") " pod="openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m" Nov 02 14:32:38.825774 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.825738 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m" Nov 02 14:32:38.983512 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:38.983487 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq"] Nov 02 14:32:38.985175 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:32:38.985132 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae683718_049c_4ea8_82da_4f97e31175ac.slice/crio-3872eac538780f89c0dfb827ba95323dbb8175ad9bc121ed4e551a7f0419bffe WatchSource:0}: Error finding container 3872eac538780f89c0dfb827ba95323dbb8175ad9bc121ed4e551a7f0419bffe: Status 404 returned error can't find the container with id 3872eac538780f89c0dfb827ba95323dbb8175ad9bc121ed4e551a7f0419bffe Nov 02 14:32:39.141679 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:39.141649 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m"] Nov 02 14:32:39.141986 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:32:39.141960 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73bd157_9369_406e_b2bf_d4cfa38a1d33.slice/crio-d52aa3eeaa4dec3fad224e4d7205eb9cd563642c41622fec5e732b9fa6a862a3 WatchSource:0}: Error finding container d52aa3eeaa4dec3fad224e4d7205eb9cd563642c41622fec5e732b9fa6a862a3: Status 404 returned error can't find the container with id d52aa3eeaa4dec3fad224e4d7205eb9cd563642c41622fec5e732b9fa6a862a3 Nov 02 14:32:39.532819 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:39.532777 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m" event={"ID":"d73bd157-9369-406e-b2bf-d4cfa38a1d33","Type":"ContainerStarted","Data":"401c5fae59a8743d91485251ffa133226e1d2a261fcbb5b6d6b3975c14a18a00"} Nov 02 14:32:39.533294 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:39.532845 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m" event={"ID":"d73bd157-9369-406e-b2bf-d4cfa38a1d33","Type":"ContainerStarted","Data":"d52aa3eeaa4dec3fad224e4d7205eb9cd563642c41622fec5e732b9fa6a862a3"} Nov 02 14:32:39.533294 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:39.532876 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m" Nov 02 14:32:39.534467 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:39.534439 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq" event={"ID":"ae683718-049c-4ea8-82da-4f97e31175ac","Type":"ContainerStarted","Data":"f2e3b6d90690af160aca9edf213c0c0a4a34e8345344ddb6dc8880a84536b0f6"} Nov 02 14:32:39.534583 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:39.534491 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq" event={"ID":"ae683718-049c-4ea8-82da-4f97e31175ac","Type":"ContainerStarted","Data":"3872eac538780f89c0dfb827ba95323dbb8175ad9bc121ed4e551a7f0419bffe"} Nov 02 14:32:39.587252 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:39.587188 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m" podStartSLOduration=1.587166826 podStartE2EDuration="1.587166826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:32:39.582482238 +0000 UTC m=+2124.766329576" watchObservedRunningTime="2026-11-02 14:32:39.587166826 +0000 UTC m=+2124.771014164" Nov 02 14:32:39.628256 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:39.628196 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq" podStartSLOduration=1.628179982 podStartE2EDuration="1.628179982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:32:39.626124011 +0000 UTC m=+2124.809971348" watchObservedRunningTime="2026-11-02 14:32:39.628179982 +0000 UTC m=+2124.812027318" Nov 02 14:32:40.541505 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:40.541471 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq" Nov 02 14:32:41.670051 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:41.670016 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84"] Nov 02 14:32:41.670491 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:41.670223 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" podUID="ebd8b16b-08fa-4933-8451-5e9660510c50" containerName="istio-proxy" containerID="cri-o://bffe2d99b14e6d2fea40598c4f4dd1e0330347abc722723c615c28e75b020cd6" gracePeriod=30 Nov 02 14:32:46.452416 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:46.452372 4866 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:32:46.452852 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:46.452440 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" podUID="ebd8b16b-08fa-4933-8451-5e9660510c50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:32:46.954830 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:46.954803 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:32:47.019899 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.019867 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd8b16b-08fa-4933-8451-5e9660510c50-kube-api-access-5rdgl\" (UniqueName: \"kubernetes.io/projected/ebd8b16b-08fa-4933-8451-5e9660510c50-kube-api-access-5rdgl\") pod \"ebd8b16b-08fa-4933-8451-5e9660510c50\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " Nov 02 14:32:47.019899 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.019902 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-podinfo\") pod \"ebd8b16b-08fa-4933-8451-5e9660510c50\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " Nov 02 14:32:47.020158 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.019968 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-credential-socket\") pod \"ebd8b16b-08fa-4933-8451-5e9660510c50\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " Nov 02 14:32:47.020158 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.020036 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-envoy\") pod \"ebd8b16b-08fa-4933-8451-5e9660510c50\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " Nov 02 14:32:47.020281 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.020192 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-workload-socket\") pod \"ebd8b16b-08fa-4933-8451-5e9660510c50\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " Nov 02 14:32:47.020281 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.020259 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-token\" (UniqueName: \"kubernetes.io/projected/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-token\") pod \"ebd8b16b-08fa-4933-8451-5e9660510c50\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " Nov 02 14:32:47.020389 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.020292 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-workload-certs\") pod \"ebd8b16b-08fa-4933-8451-5e9660510c50\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " Nov 02 14:32:47.020389 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.020358 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-data\") pod \"ebd8b16b-08fa-4933-8451-5e9660510c50\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " Nov 02 14:32:47.020497 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.020398 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebd8b16b-08fa-4933-8451-5e9660510c50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd8b16b-08fa-4933-8451-5e9660510c50-istiod-ca-cert\") pod \"ebd8b16b-08fa-4933-8451-5e9660510c50\" (UID: \"ebd8b16b-08fa-4933-8451-5e9660510c50\") " Nov 02 14:32:47.021696 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.020617 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-workload-socket" pod "ebd8b16b-08fa-4933-8451-5e9660510c50" (UID: "ebd8b16b-08fa-4933-8451-5e9660510c50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:32:47.021696 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.021046 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:32:47.021696 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.021532 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd8b16b-08fa-4933-8451-5e9660510c50-istiod-ca-cert" pod "ebd8b16b-08fa-4933-8451-5e9660510c50" (UID: "ebd8b16b-08fa-4933-8451-5e9660510c50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:32:47.021696 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.021618 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-credential-socket" pod "ebd8b16b-08fa-4933-8451-5e9660510c50" (UID: "ebd8b16b-08fa-4933-8451-5e9660510c50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:32:47.021696 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.021674 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-data" pod "ebd8b16b-08fa-4933-8451-5e9660510c50" (UID: "ebd8b16b-08fa-4933-8451-5e9660510c50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:32:47.022124 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.022014 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-workload-certs" pod "ebd8b16b-08fa-4933-8451-5e9660510c50" (UID: "ebd8b16b-08fa-4933-8451-5e9660510c50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:32:47.023449 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.023425 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd8b16b-08fa-4933-8451-5e9660510c50-kube-api-access-5rdgl" pod "ebd8b16b-08fa-4933-8451-5e9660510c50" (UID: "ebd8b16b-08fa-4933-8451-5e9660510c50"). InnerVolumeSpecName "kube-api-access-5rdgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:32:47.024145 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.024117 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-token" pod "ebd8b16b-08fa-4933-8451-5e9660510c50" (UID: "ebd8b16b-08fa-4933-8451-5e9660510c50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:32:47.028480 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.028420 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-podinfo" pod "ebd8b16b-08fa-4933-8451-5e9660510c50" (UID: "ebd8b16b-08fa-4933-8451-5e9660510c50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:32:47.028599 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.028528 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-envoy" pod "ebd8b16b-08fa-4933-8451-5e9660510c50" (UID: "ebd8b16b-08fa-4933-8451-5e9660510c50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:32:47.122230 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.122139 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:32:47.122230 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.122174 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:32:47.122230 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.122187 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:32:47.122230 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.122199 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd8b16b-08fa-4933-8451-5e9660510c50-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:32:47.122230 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.122213 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rdgl\" (UniqueName: \"kubernetes.io/projected/ebd8b16b-08fa-4933-8451-5e9660510c50-kube-api-access-5rdgl\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:32:47.122230 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.122226 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:32:47.122728 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.122240 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:32:47.122728 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.122252 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd8b16b-08fa-4933-8451-5e9660510c50-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:32:47.598245 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.598213 4866 generic.go:356] "Generic (PLEG): container finished" podID="ebd8b16b-08fa-4933-8451-5e9660510c50" containerID="bffe2d99b14e6d2fea40598c4f4dd1e0330347abc722723c615c28e75b020cd6" exitCode=0 Nov 02 14:32:47.598713 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.598264 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" Nov 02 14:32:47.598713 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.598302 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" event={"ID":"ebd8b16b-08fa-4933-8451-5e9660510c50","Type":"ContainerDied","Data":"bffe2d99b14e6d2fea40598c4f4dd1e0330347abc722723c615c28e75b020cd6"} Nov 02 14:32:47.598713 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.598359 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84" event={"ID":"ebd8b16b-08fa-4933-8451-5e9660510c50","Type":"ContainerDied","Data":"d8957374d0c97326244def51f6293f26016f1d8c5c7fc0f5d8d32249c4a167be"} Nov 02 14:32:47.598713 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.598384 4866 scope.go:122] "RemoveContainer" containerID="bffe2d99b14e6d2fea40598c4f4dd1e0330347abc722723c615c28e75b020cd6" Nov 02 14:32:47.609661 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.609640 4866 scope.go:122] "RemoveContainer" containerID="bffe2d99b14e6d2fea40598c4f4dd1e0330347abc722723c615c28e75b020cd6" Nov 02 14:32:47.609939 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:32:47.609921 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bffe2d99b14e6d2fea40598c4f4dd1e0330347abc722723c615c28e75b020cd6\": container with ID starting with bffe2d99b14e6d2fea40598c4f4dd1e0330347abc722723c615c28e75b020cd6 not found: ID does not exist" containerID="bffe2d99b14e6d2fea40598c4f4dd1e0330347abc722723c615c28e75b020cd6" Nov 02 14:32:47.610047 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.609976 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bffe2d99b14e6d2fea40598c4f4dd1e0330347abc722723c615c28e75b020cd6"} err="failed to get container status \"bffe2d99b14e6d2fea40598c4f4dd1e0330347abc722723c615c28e75b020cd6\": rpc error: code = NotFound desc = could not find container \"bffe2d99b14e6d2fea40598c4f4dd1e0330347abc722723c615c28e75b020cd6\": container with ID starting with bffe2d99b14e6d2fea40598c4f4dd1e0330347abc722723c615c28e75b020cd6 not found: ID does not exist" Nov 02 14:32:47.668027 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.667994 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84"] Nov 02 14:32:47.673248 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:47.673224 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-25r84"] Nov 02 14:32:48.961345 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:48.961305 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebd8b16b-08fa-4933-8451-5e9660510c50" path="/var/lib/kubelet/pods/ebd8b16b-08fa-4933-8451-5e9660510c50/volumes" Nov 02 14:32:49.538622 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:49.538588 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" Nov 02 14:32:50.567194 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:50.567153 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m" Nov 02 14:32:51.551390 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:32:51.551355 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq" Nov 02 14:33:04.065105 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:04.065070 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7807-0ab152"] Nov 02 14:33:04.065735 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:04.065719 4866 cpu_manager.go:414] "containerMap: removing container" podUID="ebd8b16b-08fa-4933-8451-5e9660510c50" containerName="istio-proxy" Nov 02 14:33:04.065780 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:04.065737 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebd8b16b-08fa-4933-8451-5e9660510c50" containerName="istio-proxy" Nov 02 14:33:04.065858 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:04.065847 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebd8b16b-08fa-4933-8451-5e9660510c50" containerName="istio-proxy" Nov 02 14:33:04.070933 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:04.070915 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7807-0ab152" Nov 02 14:33:04.087281 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:04.087256 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7807-0ab152"] Nov 02 14:33:04.095850 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:04.095826 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nz2z6\"" Nov 02 14:33:04.160004 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:04.159965 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbhw6\" (UniqueName: \"kubernetes.io/projected/298cc16d-2e3d-4bb2-967b-b75e1428164e-kube-api-access-sbhw6\") pod \"test-curl-7807-0ab152\" (UID: \"298cc16d-2e3d-4bb2-967b-b75e1428164e\") " pod="opendatahub/test-curl-7807-0ab152" Nov 02 14:33:04.261743 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:04.261708 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbhw6\" (UniqueName: \"kubernetes.io/projected/298cc16d-2e3d-4bb2-967b-b75e1428164e-kube-api-access-sbhw6\") pod \"test-curl-7807-0ab152\" (UID: \"298cc16d-2e3d-4bb2-967b-b75e1428164e\") " pod="opendatahub/test-curl-7807-0ab152" Nov 02 14:33:04.285647 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:04.285622 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbhw6\" (UniqueName: \"kubernetes.io/projected/298cc16d-2e3d-4bb2-967b-b75e1428164e-kube-api-access-sbhw6\") pod \"test-curl-7807-0ab152\" (UID: \"298cc16d-2e3d-4bb2-967b-b75e1428164e\") " pod="opendatahub/test-curl-7807-0ab152" Nov 02 14:33:04.383309 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:04.383225 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7807-0ab152" Nov 02 14:33:04.698838 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:04.698805 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7807-0ab152"] Nov 02 14:33:04.700358 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:33:04.700335 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod298cc16d_2e3d_4bb2_967b_b75e1428164e.slice/crio-10f628af35049a4106df6464897f27374bfdc3b41d455677d4e810f4d27e3d77 WatchSource:0}: Error finding container 10f628af35049a4106df6464897f27374bfdc3b41d455677d4e810f4d27e3d77: Status 404 returned error can't find the container with id 10f628af35049a4106df6464897f27374bfdc3b41d455677d4e810f4d27e3d77 Nov 02 14:33:04.746800 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:04.746767 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7807-0ab152" event={"ID":"298cc16d-2e3d-4bb2-967b-b75e1428164e","Type":"ContainerStarted","Data":"10f628af35049a4106df6464897f27374bfdc3b41d455677d4e810f4d27e3d77"} Nov 02 14:33:05.775004 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:05.762925 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7807-0ab152" event={"ID":"298cc16d-2e3d-4bb2-967b-b75e1428164e","Type":"ContainerStarted","Data":"073051a7e12acda10c2d9536e7ec046edea7de4606938b52fb75b55beaa2f327"} Nov 02 14:33:05.815376 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:05.812476 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7807-0ab152" podStartSLOduration=1.8124517839999998 podStartE2EDuration="1.812451784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:33:05.806265633 +0000 UTC m=+2150.990112972" watchObservedRunningTime="2026-11-02 14:33:05.812451784 +0000 UTC m=+2150.996299191" Nov 02 14:33:06.010676 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:33:06.010637 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:33:06.826973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:06.825969 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7807-0ab152"] Nov 02 14:33:06.833162 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:06.833125 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7807-0ab152"] Nov 02 14:33:07.776720 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:07.776676 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7807-0ab152" podUID="298cc16d-2e3d-4bb2-967b-b75e1428164e" containerName="test-curl-7807-0ab152" containerID="cri-o://073051a7e12acda10c2d9536e7ec046edea7de4606938b52fb75b55beaa2f327" gracePeriod=2 Nov 02 14:33:10.046300 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:10.046232 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7807-0ab152" Nov 02 14:33:10.062084 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:10.062050 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/298cc16d-2e3d-4bb2-967b-b75e1428164e-kube-api-access-sbhw6\" (UniqueName: \"kubernetes.io/projected/298cc16d-2e3d-4bb2-967b-b75e1428164e-kube-api-access-sbhw6\") pod \"298cc16d-2e3d-4bb2-967b-b75e1428164e\" (UID: \"298cc16d-2e3d-4bb2-967b-b75e1428164e\") " Nov 02 14:33:10.064280 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:10.064247 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298cc16d-2e3d-4bb2-967b-b75e1428164e-kube-api-access-sbhw6" pod "298cc16d-2e3d-4bb2-967b-b75e1428164e" (UID: "298cc16d-2e3d-4bb2-967b-b75e1428164e"). InnerVolumeSpecName "kube-api-access-sbhw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:33:10.163773 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:10.163738 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbhw6\" (UniqueName: \"kubernetes.io/projected/298cc16d-2e3d-4bb2-967b-b75e1428164e-kube-api-access-sbhw6\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:33:10.821971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:10.798488 4866 generic.go:356] "Generic (PLEG): container finished" podID="298cc16d-2e3d-4bb2-967b-b75e1428164e" containerID="073051a7e12acda10c2d9536e7ec046edea7de4606938b52fb75b55beaa2f327" exitCode=137 Nov 02 14:33:10.821971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:10.798663 4866 scope.go:122] "RemoveContainer" containerID="073051a7e12acda10c2d9536e7ec046edea7de4606938b52fb75b55beaa2f327" Nov 02 14:33:10.821971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:10.798828 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7807-0ab152" Nov 02 14:33:10.835301 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:10.835278 4866 scope.go:122] "RemoveContainer" containerID="073051a7e12acda10c2d9536e7ec046edea7de4606938b52fb75b55beaa2f327" Nov 02 14:33:10.835668 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:33:10.835643 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"073051a7e12acda10c2d9536e7ec046edea7de4606938b52fb75b55beaa2f327\": container with ID starting with 073051a7e12acda10c2d9536e7ec046edea7de4606938b52fb75b55beaa2f327 not found: ID does not exist" containerID="073051a7e12acda10c2d9536e7ec046edea7de4606938b52fb75b55beaa2f327" Nov 02 14:33:10.835733 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:10.835677 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"073051a7e12acda10c2d9536e7ec046edea7de4606938b52fb75b55beaa2f327"} err="failed to get container status \"073051a7e12acda10c2d9536e7ec046edea7de4606938b52fb75b55beaa2f327\": rpc error: code = NotFound desc = could not find container \"073051a7e12acda10c2d9536e7ec046edea7de4606938b52fb75b55beaa2f327\": container with ID starting with 073051a7e12acda10c2d9536e7ec046edea7de4606938b52fb75b55beaa2f327 not found: ID does not exist" Nov 02 14:33:10.846357 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:33:10.846329 4866 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7807-0ab152\" is forbidden: User \"system:node:ip-10-0-134-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-99.ec2.internal' and this object" podUID="298cc16d-2e3d-4bb2-967b-b75e1428164e" pod="opendatahub/test-curl-7807-0ab152" Nov 02 14:33:10.947079 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:10.947044 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="298cc16d-2e3d-4bb2-967b-b75e1428164e" path="/var/lib/kubelet/pods/298cc16d-2e3d-4bb2-967b-b75e1428164e/volumes" Nov 02 14:33:15.941341 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:15.941274 4866 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": read tcp 10.132.0.2:33880->10.132.0.131:15021: read: connection reset by peer" start-of-body= Nov 02 14:33:15.941781 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:15.941369 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" podUID="6bc409ca-3338-4474-bad2-aeb03d23aedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": read tcp 10.132.0.2:33880->10.132.0.131:15021: read: connection reset by peer" Nov 02 14:33:16.864075 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:16.864042 4866 generic.go:356] "Generic (PLEG): container finished" podID="6bc409ca-3338-4474-bad2-aeb03d23aedc" containerID="17c24af6f40d321f5fc7021d3fab583d05511ffcc6a4af356fb15f4159f89cd0" exitCode=0 Nov 02 14:33:16.864252 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:16.864110 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" event={"ID":"6bc409ca-3338-4474-bad2-aeb03d23aedc","Type":"ContainerDied","Data":"17c24af6f40d321f5fc7021d3fab583d05511ffcc6a4af356fb15f4159f89cd0"} Nov 02 14:33:16.864572 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:16.864552 4866 scope.go:122] "RemoveContainer" containerID="17c24af6f40d321f5fc7021d3fab583d05511ffcc6a4af356fb15f4159f89cd0" Nov 02 14:33:16.866121 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:16.866086 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:33:16.866189 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:16.866160 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:33:16.866227 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:16.866196 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:33:17.877284 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:17.877238 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" event={"ID":"6bc409ca-3338-4474-bad2-aeb03d23aedc","Type":"ContainerStarted","Data":"f305c6bde605dddd1576a2ebfd1a7acb02a443b2c4eeaf6f1ca40ed435648e6c"} Nov 02 14:33:18.071805 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:18.071765 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2"] Nov 02 14:33:18.125973 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:18.125896 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq"] Nov 02 14:33:18.231288 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:18.231254 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:33:18.231654 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:18.231628 4866 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 14:33:18.231868 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:18.231842 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" podUID="6bc409ca-3338-4474-bad2-aeb03d23aedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 14:33:19.225485 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:19.225453 4866 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 14:33:19.225957 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:19.225508 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" podUID="6bc409ca-3338-4474-bad2-aeb03d23aedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 14:33:20.226310 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:20.226255 4866 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 14:33:20.226811 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:20.226344 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" podUID="6bc409ca-3338-4474-bad2-aeb03d23aedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 14:33:21.228385 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:21.227714 4866 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 14:33:21.228385 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:21.227796 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" podUID="6bc409ca-3338-4474-bad2-aeb03d23aedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 14:33:22.225844 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:22.225805 4866 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 14:33:22.226046 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:22.225868 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" podUID="6bc409ca-3338-4474-bad2-aeb03d23aedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 14:33:23.072487 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:23.072440 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" podUID="65738079-7a3e-4f67-9dbc-0a1b8c067326" containerName="payload-processing" containerID="cri-o://e97c4883a9e7d3dfdd3a2857054309bacd6e3344acfdab7e6f21e5b1c8dede7f" gracePeriod=25 Nov 02 14:33:23.127237 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:23.127186 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" podUID="325d29d4-150f-4b92-8852-c7df775e9ac1" containerName="payload-pre-processing" containerID="cri-o://73dbace5328b085f0ef58ba3c789f8f335941f0a22421de33d76b3ce23458f76" gracePeriod=25 Nov 02 14:33:23.337757 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:23.337670 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:33:23.338015 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:23.337991 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:33:23.338977 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:23.338934 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:33:23.516084 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:23.516040 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" podUID="325d29d4-150f-4b92-8852-c7df775e9ac1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:33:23.519905 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:23.519868 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" podUID="65738079-7a3e-4f67-9dbc-0a1b8c067326" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:33:26.991327 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:26.991293 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q"] Nov 02 14:33:26.991764 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:26.991506 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" podUID="38537e17-d921-43b8-aa90-15a82b39342b" containerName="istio-proxy" containerID="cri-o://c27b3ae7e198db48b8d998d192d0dfb260ae118dcc30e1d46a9e8dc5c9a5416a" gracePeriod=30 Nov 02 14:33:30.348198 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.348159 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4"] Nov 02 14:33:30.349110 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.349082 4866 cpu_manager.go:414] "containerMap: removing container" podUID="298cc16d-2e3d-4bb2-967b-b75e1428164e" containerName="test-curl-7807-0ab152" Nov 02 14:33:30.349110 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.349109 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="298cc16d-2e3d-4bb2-967b-b75e1428164e" containerName="test-curl-7807-0ab152" Nov 02 14:33:30.349323 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.349279 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="298cc16d-2e3d-4bb2-967b-b75e1428164e" containerName="test-curl-7807-0ab152" Nov 02 14:33:30.353031 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.353011 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.382832 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.380891 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e07583\"/\"sub-test-model-e07583-kserve-self-signed-certs\"" Nov 02 14:33:30.382832 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.381156 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e07583\"/\"default-dockercfg-mb6lv\"" Nov 02 14:33:30.387069 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.387037 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4"] Nov 02 14:33:30.423921 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.423871 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e07583\"/\"kube-root-ca.crt\"" Nov 02 14:33:30.423921 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.423892 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e07583\"/\"openshift-service-ca.crt\"" Nov 02 14:33:30.452887 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.452850 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-dshm\") pod \"sub-test-model-e07583-kserve-58568f678f-zmnd4\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.453080 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.452897 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-home\") pod \"sub-test-model-e07583-kserve-58568f678f-zmnd4\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.453080 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.453047 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-model-cache\") pod \"sub-test-model-e07583-kserve-58568f678f-zmnd4\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.453169 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.453088 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjn8x\" (UniqueName: \"kubernetes.io/projected/844d3fe8-1341-4a06-82f0-f119401c4a6c-kube-api-access-cjn8x\") pod \"sub-test-model-e07583-kserve-58568f678f-zmnd4\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.453169 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.453143 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-tmp-dir\") pod \"sub-test-model-e07583-kserve-58568f678f-zmnd4\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.453284 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.453270 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/844d3fe8-1341-4a06-82f0-f119401c4a6c-tls-certs\") pod \"sub-test-model-e07583-kserve-58568f678f-zmnd4\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.554565 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.554521 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/844d3fe8-1341-4a06-82f0-f119401c4a6c-tls-certs\") pod \"sub-test-model-e07583-kserve-58568f678f-zmnd4\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.554773 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.554582 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-dshm\") pod \"sub-test-model-e07583-kserve-58568f678f-zmnd4\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.554773 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.554624 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-home\") pod \"sub-test-model-e07583-kserve-58568f678f-zmnd4\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.554886 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.554705 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-model-cache\") pod \"sub-test-model-e07583-kserve-58568f678f-zmnd4\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.554886 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.554856 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjn8x\" (UniqueName: \"kubernetes.io/projected/844d3fe8-1341-4a06-82f0-f119401c4a6c-kube-api-access-cjn8x\") pod \"sub-test-model-e07583-kserve-58568f678f-zmnd4\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.555027 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.554894 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-tmp-dir\") pod \"sub-test-model-e07583-kserve-58568f678f-zmnd4\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.555142 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.555124 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-home\") pod \"sub-test-model-e07583-kserve-58568f678f-zmnd4\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.555214 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.555180 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-model-cache\") pod \"sub-test-model-e07583-kserve-58568f678f-zmnd4\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.555265 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.555251 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-tmp-dir\") pod \"sub-test-model-e07583-kserve-58568f678f-zmnd4\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.557485 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.557462 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-dshm\") pod \"sub-test-model-e07583-kserve-58568f678f-zmnd4\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.557755 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.557736 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/844d3fe8-1341-4a06-82f0-f119401c4a6c-tls-certs\") pod \"sub-test-model-e07583-kserve-58568f678f-zmnd4\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.580245 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.580208 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjn8x\" (UniqueName: \"kubernetes.io/projected/844d3fe8-1341-4a06-82f0-f119401c4a6c-kube-api-access-cjn8x\") pod \"sub-test-model-e07583-kserve-58568f678f-zmnd4\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.669643 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.669557 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:30.987524 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:30.987498 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4"] Nov 02 14:33:30.988709 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:33:30.988685 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod844d3fe8_1341_4a06_82f0_f119401c4a6c.slice/crio-8af73f32dc3339985c3612ebc99cc3c8833998445ffe6cbecbf45dfeee5bc62d WatchSource:0}: Error finding container 8af73f32dc3339985c3612ebc99cc3c8833998445ffe6cbecbf45dfeee5bc62d: Status 404 returned error can't find the container with id 8af73f32dc3339985c3612ebc99cc3c8833998445ffe6cbecbf45dfeee5bc62d Nov 02 14:33:31.989763 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:31.989722 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" event={"ID":"844d3fe8-1341-4a06-82f0-f119401c4a6c","Type":"ContainerStarted","Data":"13503b7ba366e7a57736d4792dff24adb26c33072a47f4bf4c5f0a978ae8c6f2"} Nov 02 14:33:31.990297 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:31.989775 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" event={"ID":"844d3fe8-1341-4a06-82f0-f119401c4a6c","Type":"ContainerStarted","Data":"8af73f32dc3339985c3612ebc99cc3c8833998445ffe6cbecbf45dfeee5bc62d"} Nov 02 14:33:32.033191 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.033123 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" podStartSLOduration=2.033109206 podStartE2EDuration="2.033109206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:33:32.031738038 +0000 UTC m=+2177.215585376" watchObservedRunningTime="2026-11-02 14:33:32.033109206 +0000 UTC m=+2177.216956556" Nov 02 14:33:32.283271 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.283245 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:33:32.379085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.379053 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-credential-socket\") pod \"38537e17-d921-43b8-aa90-15a82b39342b\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " Nov 02 14:33:32.379266 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.379099 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38537e17-d921-43b8-aa90-15a82b39342b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38537e17-d921-43b8-aa90-15a82b39342b-istio-podinfo\") pod \"38537e17-d921-43b8-aa90-15a82b39342b\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " Nov 02 14:33:32.379266 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.379168 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-istio-data\") pod \"38537e17-d921-43b8-aa90-15a82b39342b\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " Nov 02 14:33:32.379266 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.379190 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38537e17-d921-43b8-aa90-15a82b39342b-istio-token\" (UniqueName: \"kubernetes.io/projected/38537e17-d921-43b8-aa90-15a82b39342b-istio-token\") pod \"38537e17-d921-43b8-aa90-15a82b39342b\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " Nov 02 14:33:32.379266 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.379227 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38537e17-d921-43b8-aa90-15a82b39342b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38537e17-d921-43b8-aa90-15a82b39342b-istiod-ca-cert\") pod \"38537e17-d921-43b8-aa90-15a82b39342b\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " Nov 02 14:33:32.379474 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.379309 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-istio-envoy\") pod \"38537e17-d921-43b8-aa90-15a82b39342b\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " Nov 02 14:33:32.379474 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.379345 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-credential-socket" pod "38537e17-d921-43b8-aa90-15a82b39342b" (UID: "38537e17-d921-43b8-aa90-15a82b39342b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:33:32.379474 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.379428 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-workload-certs\") pod \"38537e17-d921-43b8-aa90-15a82b39342b\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " Nov 02 14:33:32.379474 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.379462 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38537e17-d921-43b8-aa90-15a82b39342b-kube-api-access-cprfx\" (UniqueName: \"kubernetes.io/projected/38537e17-d921-43b8-aa90-15a82b39342b-kube-api-access-cprfx\") pod \"38537e17-d921-43b8-aa90-15a82b39342b\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " Nov 02 14:33:32.379474 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.379464 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-istio-data" pod "38537e17-d921-43b8-aa90-15a82b39342b" (UID: "38537e17-d921-43b8-aa90-15a82b39342b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:33:32.379743 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.379499 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-workload-socket\") pod \"38537e17-d921-43b8-aa90-15a82b39342b\" (UID: \"38537e17-d921-43b8-aa90-15a82b39342b\") " Nov 02 14:33:32.379743 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.379692 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-workload-certs" pod "38537e17-d921-43b8-aa90-15a82b39342b" (UID: "38537e17-d921-43b8-aa90-15a82b39342b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:33:32.379743 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.379723 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38537e17-d921-43b8-aa90-15a82b39342b-istiod-ca-cert" pod "38537e17-d921-43b8-aa90-15a82b39342b" (UID: "38537e17-d921-43b8-aa90-15a82b39342b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:33:32.379930 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.379903 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-workload-socket" pod "38537e17-d921-43b8-aa90-15a82b39342b" (UID: "38537e17-d921-43b8-aa90-15a82b39342b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:33:32.380167 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.380141 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:33:32.380167 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.380170 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38537e17-d921-43b8-aa90-15a82b39342b-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:33:32.380367 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.380183 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:33:32.380367 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.380196 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:33:32.380367 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.380210 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:33:32.381535 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.381513 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-istio-envoy" pod "38537e17-d921-43b8-aa90-15a82b39342b" (UID: "38537e17-d921-43b8-aa90-15a82b39342b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:33:32.381792 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.381770 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38537e17-d921-43b8-aa90-15a82b39342b-istio-podinfo" pod "38537e17-d921-43b8-aa90-15a82b39342b" (UID: "38537e17-d921-43b8-aa90-15a82b39342b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:33:32.381926 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.381909 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38537e17-d921-43b8-aa90-15a82b39342b-kube-api-access-cprfx" pod "38537e17-d921-43b8-aa90-15a82b39342b" (UID: "38537e17-d921-43b8-aa90-15a82b39342b"). InnerVolumeSpecName "kube-api-access-cprfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:33:32.382023 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.381923 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38537e17-d921-43b8-aa90-15a82b39342b-istio-token" pod "38537e17-d921-43b8-aa90-15a82b39342b" (UID: "38537e17-d921-43b8-aa90-15a82b39342b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:33:32.480959 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.480913 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38537e17-d921-43b8-aa90-15a82b39342b-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:33:32.480959 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.480955 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cprfx\" (UniqueName: \"kubernetes.io/projected/38537e17-d921-43b8-aa90-15a82b39342b-kube-api-access-cprfx\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:33:32.481213 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.480970 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38537e17-d921-43b8-aa90-15a82b39342b-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:33:32.481213 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.480980 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38537e17-d921-43b8-aa90-15a82b39342b-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:33:32.669678 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.669590 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:32.998667 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.998630 4866 generic.go:356] "Generic (PLEG): container finished" podID="38537e17-d921-43b8-aa90-15a82b39342b" containerID="c27b3ae7e198db48b8d998d192d0dfb260ae118dcc30e1d46a9e8dc5c9a5416a" exitCode=0 Nov 02 14:33:32.999155 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.998752 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" Nov 02 14:33:32.999155 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.998812 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" event={"ID":"38537e17-d921-43b8-aa90-15a82b39342b","Type":"ContainerDied","Data":"c27b3ae7e198db48b8d998d192d0dfb260ae118dcc30e1d46a9e8dc5c9a5416a"} Nov 02 14:33:32.999155 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.998850 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q" event={"ID":"38537e17-d921-43b8-aa90-15a82b39342b","Type":"ContainerDied","Data":"7a13919aa1638de25109329396c2aa8b3b8c01ab80f6288fb71b536bc0ca0130"} Nov 02 14:33:32.999155 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:32.998871 4866 scope.go:122] "RemoveContainer" containerID="c27b3ae7e198db48b8d998d192d0dfb260ae118dcc30e1d46a9e8dc5c9a5416a" Nov 02 14:33:33.009343 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:33.009318 4866 scope.go:122] "RemoveContainer" containerID="5e15c0dd5dbc9280f3bae13a30f10045b2003fa01649f000771874e730d897c9" Nov 02 14:33:33.018739 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:33.018719 4866 scope.go:122] "RemoveContainer" containerID="c27b3ae7e198db48b8d998d192d0dfb260ae118dcc30e1d46a9e8dc5c9a5416a" Nov 02 14:33:33.019038 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:33:33.019021 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c27b3ae7e198db48b8d998d192d0dfb260ae118dcc30e1d46a9e8dc5c9a5416a\": container with ID starting with c27b3ae7e198db48b8d998d192d0dfb260ae118dcc30e1d46a9e8dc5c9a5416a not found: ID does not exist" containerID="c27b3ae7e198db48b8d998d192d0dfb260ae118dcc30e1d46a9e8dc5c9a5416a" Nov 02 14:33:33.019109 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:33.019046 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27b3ae7e198db48b8d998d192d0dfb260ae118dcc30e1d46a9e8dc5c9a5416a"} err="failed to get container status \"c27b3ae7e198db48b8d998d192d0dfb260ae118dcc30e1d46a9e8dc5c9a5416a\": rpc error: code = NotFound desc = could not find container \"c27b3ae7e198db48b8d998d192d0dfb260ae118dcc30e1d46a9e8dc5c9a5416a\": container with ID starting with c27b3ae7e198db48b8d998d192d0dfb260ae118dcc30e1d46a9e8dc5c9a5416a not found: ID does not exist" Nov 02 14:33:33.019109 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:33.019066 4866 scope.go:122] "RemoveContainer" containerID="5e15c0dd5dbc9280f3bae13a30f10045b2003fa01649f000771874e730d897c9" Nov 02 14:33:33.019340 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:33:33.019324 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e15c0dd5dbc9280f3bae13a30f10045b2003fa01649f000771874e730d897c9\": container with ID starting with 5e15c0dd5dbc9280f3bae13a30f10045b2003fa01649f000771874e730d897c9 not found: ID does not exist" containerID="5e15c0dd5dbc9280f3bae13a30f10045b2003fa01649f000771874e730d897c9" Nov 02 14:33:33.019389 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:33.019343 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e15c0dd5dbc9280f3bae13a30f10045b2003fa01649f000771874e730d897c9"} err="failed to get container status \"5e15c0dd5dbc9280f3bae13a30f10045b2003fa01649f000771874e730d897c9\": rpc error: code = NotFound desc = could not find container \"5e15c0dd5dbc9280f3bae13a30f10045b2003fa01649f000771874e730d897c9\": container with ID starting with 5e15c0dd5dbc9280f3bae13a30f10045b2003fa01649f000771874e730d897c9 not found: ID does not exist" Nov 02 14:33:33.064056 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:33.064023 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q"] Nov 02 14:33:33.075500 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:33.075444 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-47p6q"] Nov 02 14:33:33.515598 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:33.515556 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" podUID="325d29d4-150f-4b92-8852-c7df775e9ac1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:33:33.519279 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:33.519240 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" podUID="65738079-7a3e-4f67-9dbc-0a1b8c067326" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:33:34.946747 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:34.946715 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38537e17-d921-43b8-aa90-15a82b39342b" path="/var/lib/kubelet/pods/38537e17-d921-43b8-aa90-15a82b39342b/volumes" Nov 02 14:33:36.010783 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:33:36.010736 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:33:40.670026 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:40.669991 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:40.684148 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:40.684120 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:41.063158 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:41.063126 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:33:43.404629 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:43.404594 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" Nov 02 14:33:43.446573 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:43.446542 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" Nov 02 14:33:43.516630 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:43.516593 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/325d29d4-150f-4b92-8852-c7df775e9ac1-kube-api-access-smlkd\" (UniqueName: \"kubernetes.io/projected/325d29d4-150f-4b92-8852-c7df775e9ac1-kube-api-access-smlkd\") pod \"325d29d4-150f-4b92-8852-c7df775e9ac1\" (UID: \"325d29d4-150f-4b92-8852-c7df775e9ac1\") " Nov 02 14:33:43.516825 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:43.516776 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65738079-7a3e-4f67-9dbc-0a1b8c067326-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65738079-7a3e-4f67-9dbc-0a1b8c067326-plugins-config-volume\") pod \"65738079-7a3e-4f67-9dbc-0a1b8c067326\" (UID: \"65738079-7a3e-4f67-9dbc-0a1b8c067326\") " Nov 02 14:33:43.516825 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:43.516812 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/325d29d4-150f-4b92-8852-c7df775e9ac1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325d29d4-150f-4b92-8852-c7df775e9ac1-plugins-config-volume\") pod \"325d29d4-150f-4b92-8852-c7df775e9ac1\" (UID: \"325d29d4-150f-4b92-8852-c7df775e9ac1\") " Nov 02 14:33:43.516965 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:43.516840 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65738079-7a3e-4f67-9dbc-0a1b8c067326-kube-api-access-cjq6q\" (UniqueName: \"kubernetes.io/projected/65738079-7a3e-4f67-9dbc-0a1b8c067326-kube-api-access-cjq6q\") pod \"65738079-7a3e-4f67-9dbc-0a1b8c067326\" (UID: \"65738079-7a3e-4f67-9dbc-0a1b8c067326\") " Nov 02 14:33:43.517282 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:43.517254 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/325d29d4-150f-4b92-8852-c7df775e9ac1-plugins-config-volume" pod "325d29d4-150f-4b92-8852-c7df775e9ac1" (UID: "325d29d4-150f-4b92-8852-c7df775e9ac1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:33:43.517360 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:43.517252 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65738079-7a3e-4f67-9dbc-0a1b8c067326-plugins-config-volume" pod "65738079-7a3e-4f67-9dbc-0a1b8c067326" (UID: "65738079-7a3e-4f67-9dbc-0a1b8c067326"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:33:43.517511 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:43.517492 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65738079-7a3e-4f67-9dbc-0a1b8c067326-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:33:43.517572 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:43.517516 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325d29d4-150f-4b92-8852-c7df775e9ac1-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:33:43.519245 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:43.519222 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65738079-7a3e-4f67-9dbc-0a1b8c067326-kube-api-access-cjq6q" pod "65738079-7a3e-4f67-9dbc-0a1b8c067326" (UID: "65738079-7a3e-4f67-9dbc-0a1b8c067326"). InnerVolumeSpecName "kube-api-access-cjq6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:33:43.519828 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:43.519804 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325d29d4-150f-4b92-8852-c7df775e9ac1-kube-api-access-smlkd" pod "325d29d4-150f-4b92-8852-c7df775e9ac1" (UID: "325d29d4-150f-4b92-8852-c7df775e9ac1"). InnerVolumeSpecName "kube-api-access-smlkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:33:43.619432 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:43.619347 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smlkd\" (UniqueName: \"kubernetes.io/projected/325d29d4-150f-4b92-8852-c7df775e9ac1-kube-api-access-smlkd\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:33:43.619432 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:43.619378 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjq6q\" (UniqueName: \"kubernetes.io/projected/65738079-7a3e-4f67-9dbc-0a1b8c067326-kube-api-access-cjq6q\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:33:44.072414 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.072376 4866 generic.go:356] "Generic (PLEG): container finished" podID="65738079-7a3e-4f67-9dbc-0a1b8c067326" containerID="e97c4883a9e7d3dfdd3a2857054309bacd6e3344acfdab7e6f21e5b1c8dede7f" exitCode=0 Nov 02 14:33:44.072621 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.072440 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" Nov 02 14:33:44.072621 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.072458 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" event={"ID":"65738079-7a3e-4f67-9dbc-0a1b8c067326","Type":"ContainerDied","Data":"e97c4883a9e7d3dfdd3a2857054309bacd6e3344acfdab7e6f21e5b1c8dede7f"} Nov 02 14:33:44.072621 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.072522 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2" event={"ID":"65738079-7a3e-4f67-9dbc-0a1b8c067326","Type":"ContainerDied","Data":"694ad87251150f0fb1ac907140a88c0d3cebc1c82ed6a9eb871bfb2cd1f2f3a4"} Nov 02 14:33:44.072621 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.072535 4866 scope.go:122] "RemoveContainer" containerID="e97c4883a9e7d3dfdd3a2857054309bacd6e3344acfdab7e6f21e5b1c8dede7f" Nov 02 14:33:44.074263 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.074228 4866 generic.go:356] "Generic (PLEG): container finished" podID="325d29d4-150f-4b92-8852-c7df775e9ac1" containerID="73dbace5328b085f0ef58ba3c789f8f335941f0a22421de33d76b3ce23458f76" exitCode=0 Nov 02 14:33:44.074378 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.074311 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" Nov 02 14:33:44.074575 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.074313 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" event={"ID":"325d29d4-150f-4b92-8852-c7df775e9ac1","Type":"ContainerDied","Data":"73dbace5328b085f0ef58ba3c789f8f335941f0a22421de33d76b3ce23458f76"} Nov 02 14:33:44.074701 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.074606 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq" event={"ID":"325d29d4-150f-4b92-8852-c7df775e9ac1","Type":"ContainerDied","Data":"55e4d870e1a436f79a46edd5033b9062d36b840baa895b2598a1fcce5c203921"} Nov 02 14:33:44.091221 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.091197 4866 scope.go:122] "RemoveContainer" containerID="e97c4883a9e7d3dfdd3a2857054309bacd6e3344acfdab7e6f21e5b1c8dede7f" Nov 02 14:33:44.091578 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:33:44.091551 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e97c4883a9e7d3dfdd3a2857054309bacd6e3344acfdab7e6f21e5b1c8dede7f\": container with ID starting with e97c4883a9e7d3dfdd3a2857054309bacd6e3344acfdab7e6f21e5b1c8dede7f not found: ID does not exist" containerID="e97c4883a9e7d3dfdd3a2857054309bacd6e3344acfdab7e6f21e5b1c8dede7f" Nov 02 14:33:44.091723 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.091589 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e97c4883a9e7d3dfdd3a2857054309bacd6e3344acfdab7e6f21e5b1c8dede7f"} err="failed to get container status \"e97c4883a9e7d3dfdd3a2857054309bacd6e3344acfdab7e6f21e5b1c8dede7f\": rpc error: code = NotFound desc = could not find container \"e97c4883a9e7d3dfdd3a2857054309bacd6e3344acfdab7e6f21e5b1c8dede7f\": container with ID starting with e97c4883a9e7d3dfdd3a2857054309bacd6e3344acfdab7e6f21e5b1c8dede7f not found: ID does not exist" Nov 02 14:33:44.091723 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.091615 4866 scope.go:122] "RemoveContainer" containerID="73dbace5328b085f0ef58ba3c789f8f335941f0a22421de33d76b3ce23458f76" Nov 02 14:33:44.108841 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.108814 4866 scope.go:122] "RemoveContainer" containerID="73dbace5328b085f0ef58ba3c789f8f335941f0a22421de33d76b3ce23458f76" Nov 02 14:33:44.109276 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:33:44.109252 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73dbace5328b085f0ef58ba3c789f8f335941f0a22421de33d76b3ce23458f76\": container with ID starting with 73dbace5328b085f0ef58ba3c789f8f335941f0a22421de33d76b3ce23458f76 not found: ID does not exist" containerID="73dbace5328b085f0ef58ba3c789f8f335941f0a22421de33d76b3ce23458f76" Nov 02 14:33:44.109382 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.109287 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73dbace5328b085f0ef58ba3c789f8f335941f0a22421de33d76b3ce23458f76"} err="failed to get container status \"73dbace5328b085f0ef58ba3c789f8f335941f0a22421de33d76b3ce23458f76\": rpc error: code = NotFound desc = could not find container \"73dbace5328b085f0ef58ba3c789f8f335941f0a22421de33d76b3ce23458f76\": container with ID starting with 73dbace5328b085f0ef58ba3c789f8f335941f0a22421de33d76b3ce23458f76 not found: ID does not exist" Nov 02 14:33:44.144183 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.144143 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq"] Nov 02 14:33:44.156382 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.156347 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-mkwdq"] Nov 02 14:33:44.185258 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.185221 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2"] Nov 02 14:33:44.195147 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.195111 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-29vx2"] Nov 02 14:33:44.950769 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.950731 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="325d29d4-150f-4b92-8852-c7df775e9ac1" path="/var/lib/kubelet/pods/325d29d4-150f-4b92-8852-c7df775e9ac1/volumes" Nov 02 14:33:44.951339 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:44.951321 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65738079-7a3e-4f67-9dbc-0a1b8c067326" path="/var/lib/kubelet/pods/65738079-7a3e-4f67-9dbc-0a1b8c067326/volumes" Nov 02 14:33:49.697101 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.697066 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf"] Nov 02 14:33:49.697659 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.697645 4866 cpu_manager.go:414] "containerMap: removing container" podUID="325d29d4-150f-4b92-8852-c7df775e9ac1" containerName="payload-pre-processing" Nov 02 14:33:49.697659 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.697659 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="325d29d4-150f-4b92-8852-c7df775e9ac1" containerName="payload-pre-processing" Nov 02 14:33:49.697761 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.697711 4866 cpu_manager.go:414] "containerMap: removing container" podUID="38537e17-d921-43b8-aa90-15a82b39342b" containerName="istio-proxy" Nov 02 14:33:49.697761 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.697717 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38537e17-d921-43b8-aa90-15a82b39342b" containerName="istio-proxy" Nov 02 14:33:49.697761 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.697725 4866 cpu_manager.go:414] "containerMap: removing container" podUID="38537e17-d921-43b8-aa90-15a82b39342b" containerName="istio-proxy" Nov 02 14:33:49.697761 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.697730 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38537e17-d921-43b8-aa90-15a82b39342b" containerName="istio-proxy" Nov 02 14:33:49.697761 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.697741 4866 cpu_manager.go:414] "containerMap: removing container" podUID="65738079-7a3e-4f67-9dbc-0a1b8c067326" containerName="payload-processing" Nov 02 14:33:49.697761 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.697748 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65738079-7a3e-4f67-9dbc-0a1b8c067326" containerName="payload-processing" Nov 02 14:33:49.697969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.697823 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="38537e17-d921-43b8-aa90-15a82b39342b" containerName="istio-proxy" Nov 02 14:33:49.697969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.697833 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="65738079-7a3e-4f67-9dbc-0a1b8c067326" containerName="payload-processing" Nov 02 14:33:49.697969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.697841 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="325d29d4-150f-4b92-8852-c7df775e9ac1" containerName="payload-pre-processing" Nov 02 14:33:49.702919 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.702900 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" Nov 02 14:33:49.745001 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.739077 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"agc-cred-9b6356093816\"" Nov 02 14:33:49.745001 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.739133 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"kube-root-ca.crt\"" Nov 02 14:33:49.745001 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.739183 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"openshift-service-ca.crt\"" Nov 02 14:33:49.745001 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.739142 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"payload-processing-external-model-e2e--3ffe20a0-dockercfg-7l9pt\"" Nov 02 14:33:49.745001 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.739249 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"payload-processing-external-model-plugins-e2e-extmod-b-12b8da\"" Nov 02 14:33:49.745001 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.743074 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"routing-overlay\"" Nov 02 14:33:49.745001 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.744145 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf"] Nov 02 14:33:49.806977 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.806776 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lbsd\" (UniqueName: \"kubernetes.io/projected/f06bd94b-7a9c-4463-8d81-1650b45cafdc-kube-api-access-9lbsd\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf\" (UID: \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" Nov 02 14:33:49.806977 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.806920 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f06bd94b-7a9c-4463-8d81-1650b45cafdc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf\" (UID: \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" Nov 02 14:33:49.806977 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.806981 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06bd94b-7a9c-4463-8d81-1650b45cafdc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf\" (UID: \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" Nov 02 14:33:49.807300 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.807058 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f06bd94b-7a9c-4463-8d81-1650b45cafdc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf\" (UID: \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" Nov 02 14:33:49.908193 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.908156 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f06bd94b-7a9c-4463-8d81-1650b45cafdc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf\" (UID: \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" Nov 02 14:33:49.908193 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.908199 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06bd94b-7a9c-4463-8d81-1650b45cafdc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf\" (UID: \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" Nov 02 14:33:49.908448 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.908251 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f06bd94b-7a9c-4463-8d81-1650b45cafdc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf\" (UID: \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" Nov 02 14:33:49.908448 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.908334 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lbsd\" (UniqueName: \"kubernetes.io/projected/f06bd94b-7a9c-4463-8d81-1650b45cafdc-kube-api-access-9lbsd\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf\" (UID: \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" Nov 02 14:33:49.909034 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.909006 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f06bd94b-7a9c-4463-8d81-1650b45cafdc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf\" (UID: \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" Nov 02 14:33:49.909164 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.909142 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06bd94b-7a9c-4463-8d81-1650b45cafdc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf\" (UID: \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" Nov 02 14:33:49.911078 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.910973 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f06bd94b-7a9c-4463-8d81-1650b45cafdc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf\" (UID: \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" Nov 02 14:33:49.936450 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:49.936413 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lbsd\" (UniqueName: \"kubernetes.io/projected/f06bd94b-7a9c-4463-8d81-1650b45cafdc-kube-api-access-9lbsd\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf\" (UID: \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" Nov 02 14:33:50.017116 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:50.017025 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" Nov 02 14:33:50.327980 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:50.327931 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf"] Nov 02 14:33:50.329794 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:33:50.329769 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06bd94b_7a9c_4463_8d81_1650b45cafdc.slice/crio-72b0746114c13d9a95de6f40912784cbfd4fbdea532e85cec6f7060ef4042e5d WatchSource:0}: Error finding container 72b0746114c13d9a95de6f40912784cbfd4fbdea532e85cec6f7060ef4042e5d: Status 404 returned error can't find the container with id 72b0746114c13d9a95de6f40912784cbfd4fbdea532e85cec6f7060ef4042e5d Nov 02 14:33:51.125758 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:51.125724 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" event={"ID":"f06bd94b-7a9c-4463-8d81-1650b45cafdc","Type":"ContainerStarted","Data":"af1ae0442c88c59475cb4dcd62b7aa56b25dcd590d3c803c17cee03085b3ad8f"} Nov 02 14:33:51.126201 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:51.125783 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" event={"ID":"f06bd94b-7a9c-4463-8d81-1650b45cafdc","Type":"ContainerStarted","Data":"72b0746114c13d9a95de6f40912784cbfd4fbdea532e85cec6f7060ef4042e5d"} Nov 02 14:33:51.126201 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:51.125806 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" Nov 02 14:33:51.166941 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:51.166884 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" podStartSLOduration=2.166869343 podStartE2EDuration="2.166869343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:33:51.166600432 +0000 UTC m=+2196.350447768" watchObservedRunningTime="2026-11-02 14:33:51.166869343 +0000 UTC m=+2196.350716679" Nov 02 14:33:58.743983 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.743884 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth"] Nov 02 14:33:58.744837 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.744812 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="38537e17-d921-43b8-aa90-15a82b39342b" containerName="istio-proxy" Nov 02 14:33:58.748579 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.748554 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:58.778599 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.778567 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5ba53f\"/\"openshift-service-ca.crt\"" Nov 02 14:33:58.778795 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.778630 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5ba53f\"/\"sub-test-model-5ba53f-kserve-self-signed-certs\"" Nov 02 14:33:58.778795 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.778677 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5ba53f\"/\"default-dockercfg-tflbl\"" Nov 02 14:33:58.778795 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.778759 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5ba53f\"/\"kube-root-ca.crt\"" Nov 02 14:33:58.781781 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.781750 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth"] Nov 02 14:33:58.822399 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.822365 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-home\") pod \"sub-test-model-5ba53f-kserve-56dc78469-d7kth\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:58.822399 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.822404 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-dshm\") pod \"sub-test-model-5ba53f-kserve-56dc78469-d7kth\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:58.822651 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.822494 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd2kd\" (UniqueName: \"kubernetes.io/projected/a76dbed2-c500-4d2e-8073-8b0936caecaf-kube-api-access-jd2kd\") pod \"sub-test-model-5ba53f-kserve-56dc78469-d7kth\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:58.822651 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.822619 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-model-cache\") pod \"sub-test-model-5ba53f-kserve-56dc78469-d7kth\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:58.822769 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.822662 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-tmp-dir\") pod \"sub-test-model-5ba53f-kserve-56dc78469-d7kth\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:58.822825 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.822781 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76dbed2-c500-4d2e-8073-8b0936caecaf-tls-certs\") pod \"sub-test-model-5ba53f-kserve-56dc78469-d7kth\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:58.924231 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.924189 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76dbed2-c500-4d2e-8073-8b0936caecaf-tls-certs\") pod \"sub-test-model-5ba53f-kserve-56dc78469-d7kth\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:58.924418 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.924340 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-home\") pod \"sub-test-model-5ba53f-kserve-56dc78469-d7kth\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:58.924418 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.924374 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-dshm\") pod \"sub-test-model-5ba53f-kserve-56dc78469-d7kth\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:58.924540 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.924433 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd2kd\" (UniqueName: \"kubernetes.io/projected/a76dbed2-c500-4d2e-8073-8b0936caecaf-kube-api-access-jd2kd\") pod \"sub-test-model-5ba53f-kserve-56dc78469-d7kth\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:58.924540 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.924508 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-model-cache\") pod \"sub-test-model-5ba53f-kserve-56dc78469-d7kth\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:58.924996 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.924755 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-tmp-dir\") pod \"sub-test-model-5ba53f-kserve-56dc78469-d7kth\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:58.924996 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.924883 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-home\") pod \"sub-test-model-5ba53f-kserve-56dc78469-d7kth\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:58.924996 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.924905 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-model-cache\") pod \"sub-test-model-5ba53f-kserve-56dc78469-d7kth\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:58.925289 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.925214 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-tmp-dir\") pod \"sub-test-model-5ba53f-kserve-56dc78469-d7kth\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:58.927516 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.927481 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76dbed2-c500-4d2e-8073-8b0936caecaf-tls-certs\") pod \"sub-test-model-5ba53f-kserve-56dc78469-d7kth\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:58.927516 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.927500 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-dshm\") pod \"sub-test-model-5ba53f-kserve-56dc78469-d7kth\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:58.964470 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:58.964427 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd2kd\" (UniqueName: \"kubernetes.io/projected/a76dbed2-c500-4d2e-8073-8b0936caecaf-kube-api-access-jd2kd\") pod \"sub-test-model-5ba53f-kserve-56dc78469-d7kth\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:59.063242 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:59.063156 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:33:59.389230 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:33:59.389195 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth"] Nov 02 14:33:59.392406 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:33:59.392376 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda76dbed2_c500_4d2e_8073_8b0936caecaf.slice/crio-28187ca116e433f37e1047f600dac30d1adbc271f4e2aa38f811b90c8829a1c1 WatchSource:0}: Error finding container 28187ca116e433f37e1047f600dac30d1adbc271f4e2aa38f811b90c8829a1c1: Status 404 returned error can't find the container with id 28187ca116e433f37e1047f600dac30d1adbc271f4e2aa38f811b90c8829a1c1 Nov 02 14:34:00.195927 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:00.195887 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" event={"ID":"a76dbed2-c500-4d2e-8073-8b0936caecaf","Type":"ContainerStarted","Data":"89dab39099ef576ac8f5aaad9200e25ba5873874ec4019152c034e9c4d722537"} Nov 02 14:34:00.196368 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:00.195955 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" event={"ID":"a76dbed2-c500-4d2e-8073-8b0936caecaf","Type":"ContainerStarted","Data":"28187ca116e433f37e1047f600dac30d1adbc271f4e2aa38f811b90c8829a1c1"} Nov 02 14:34:00.239488 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:00.239421 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" podStartSLOduration=2.239400406 podStartE2EDuration="2.239400406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:34:00.23754753 +0000 UTC m=+2205.421394868" watchObservedRunningTime="2026-11-02 14:34:00.239400406 +0000 UTC m=+2205.423247746" Nov 02 14:34:01.063832 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:01.063787 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:34:02.149433 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:02.149404 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" Nov 02 14:34:06.011084 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:34:06.011048 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:34:06.545057 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:06.545019 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w"] Nov 02 14:34:06.545353 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:06.545293 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" podUID="186b144a-c571-47a5-8ca0-264b72fcaea4" containerName="maas-api" containerID="cri-o://1374fdacd06cd25e1f36197f7e0ca91c50193c5e4dc1fc31a2a9559a82596164" gracePeriod=30 Nov 02 14:34:06.833092 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:06.826868 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" Nov 02 14:34:06.935437 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:06.935403 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186b144a-c571-47a5-8ca0-264b72fcaea4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/186b144a-c571-47a5-8ca0-264b72fcaea4-metrics-certs\") pod \"186b144a-c571-47a5-8ca0-264b72fcaea4\" (UID: \"186b144a-c571-47a5-8ca0-264b72fcaea4\") " Nov 02 14:34:06.935645 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:06.935498 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186b144a-c571-47a5-8ca0-264b72fcaea4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186b144a-c571-47a5-8ca0-264b72fcaea4-openshift-service-ca\") pod \"186b144a-c571-47a5-8ca0-264b72fcaea4\" (UID: \"186b144a-c571-47a5-8ca0-264b72fcaea4\") " Nov 02 14:34:06.935645 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:06.935527 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186b144a-c571-47a5-8ca0-264b72fcaea4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186b144a-c571-47a5-8ca0-264b72fcaea4-maas-api-tls\") pod \"186b144a-c571-47a5-8ca0-264b72fcaea4\" (UID: \"186b144a-c571-47a5-8ca0-264b72fcaea4\") " Nov 02 14:34:06.935645 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:06.935580 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186b144a-c571-47a5-8ca0-264b72fcaea4-kube-api-access-df8nb\" (UniqueName: \"kubernetes.io/projected/186b144a-c571-47a5-8ca0-264b72fcaea4-kube-api-access-df8nb\") pod \"186b144a-c571-47a5-8ca0-264b72fcaea4\" (UID: \"186b144a-c571-47a5-8ca0-264b72fcaea4\") " Nov 02 14:34:06.935998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:06.935881 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186b144a-c571-47a5-8ca0-264b72fcaea4-openshift-service-ca" pod "186b144a-c571-47a5-8ca0-264b72fcaea4" (UID: "186b144a-c571-47a5-8ca0-264b72fcaea4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:34:06.936186 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:06.936160 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186b144a-c571-47a5-8ca0-264b72fcaea4-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:34:06.938128 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:06.938097 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186b144a-c571-47a5-8ca0-264b72fcaea4-kube-api-access-df8nb" pod "186b144a-c571-47a5-8ca0-264b72fcaea4" (UID: "186b144a-c571-47a5-8ca0-264b72fcaea4"). InnerVolumeSpecName "kube-api-access-df8nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:34:06.948975 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:06.938588 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186b144a-c571-47a5-8ca0-264b72fcaea4-metrics-certs" pod "186b144a-c571-47a5-8ca0-264b72fcaea4" (UID: "186b144a-c571-47a5-8ca0-264b72fcaea4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:34:06.948975 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:06.946399 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186b144a-c571-47a5-8ca0-264b72fcaea4-maas-api-tls" pod "186b144a-c571-47a5-8ca0-264b72fcaea4" (UID: "186b144a-c571-47a5-8ca0-264b72fcaea4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:34:07.037619 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:07.037585 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186b144a-c571-47a5-8ca0-264b72fcaea4-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:34:07.037619 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:07.037620 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186b144a-c571-47a5-8ca0-264b72fcaea4-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:34:07.038074 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:07.037637 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df8nb\" (UniqueName: \"kubernetes.io/projected/186b144a-c571-47a5-8ca0-264b72fcaea4-kube-api-access-df8nb\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:34:07.251970 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:07.247632 4866 generic.go:356] "Generic (PLEG): container finished" podID="186b144a-c571-47a5-8ca0-264b72fcaea4" containerID="1374fdacd06cd25e1f36197f7e0ca91c50193c5e4dc1fc31a2a9559a82596164" exitCode=0 Nov 02 14:34:07.251970 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:07.247691 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" event={"ID":"186b144a-c571-47a5-8ca0-264b72fcaea4","Type":"ContainerDied","Data":"1374fdacd06cd25e1f36197f7e0ca91c50193c5e4dc1fc31a2a9559a82596164"} Nov 02 14:34:07.251970 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:07.247757 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" event={"ID":"186b144a-c571-47a5-8ca0-264b72fcaea4","Type":"ContainerDied","Data":"4de41038c023099468bf177dad440f1d59959d1aa9ef2218daa047ac1861f7cf"} Nov 02 14:34:07.251970 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:07.247786 4866 scope.go:122] "RemoveContainer" containerID="1374fdacd06cd25e1f36197f7e0ca91c50193c5e4dc1fc31a2a9559a82596164" Nov 02 14:34:07.251970 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:07.247988 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" Nov 02 14:34:07.267157 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:07.265067 4866 scope.go:122] "RemoveContainer" containerID="1374fdacd06cd25e1f36197f7e0ca91c50193c5e4dc1fc31a2a9559a82596164" Nov 02 14:34:07.267157 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:34:07.265488 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1374fdacd06cd25e1f36197f7e0ca91c50193c5e4dc1fc31a2a9559a82596164\": container with ID starting with 1374fdacd06cd25e1f36197f7e0ca91c50193c5e4dc1fc31a2a9559a82596164 not found: ID does not exist" containerID="1374fdacd06cd25e1f36197f7e0ca91c50193c5e4dc1fc31a2a9559a82596164" Nov 02 14:34:07.267157 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:07.265521 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1374fdacd06cd25e1f36197f7e0ca91c50193c5e4dc1fc31a2a9559a82596164"} err="failed to get container status \"1374fdacd06cd25e1f36197f7e0ca91c50193c5e4dc1fc31a2a9559a82596164\": rpc error: code = NotFound desc = could not find container \"1374fdacd06cd25e1f36197f7e0ca91c50193c5e4dc1fc31a2a9559a82596164\": container with ID starting with 1374fdacd06cd25e1f36197f7e0ca91c50193c5e4dc1fc31a2a9559a82596164 not found: ID does not exist" Nov 02 14:34:07.310479 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:07.310440 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w"] Nov 02 14:34:07.319538 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:07.319505 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w"] Nov 02 14:34:08.948083 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:08.948046 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186b144a-c571-47a5-8ca0-264b72fcaea4" path="/var/lib/kubelet/pods/186b144a-c571-47a5-8ca0-264b72fcaea4/volumes" Nov 02 14:34:09.063657 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:09.063621 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:34:09.078588 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:09.078555 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:34:09.272276 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:09.272197 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:34:09.731727 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:09.731667 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-rs99w" podUID="186b144a-c571-47a5-8ca0-264b72fcaea4" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.127:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:34:16.414343 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.414248 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw"] Nov 02 14:34:16.415129 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.415106 4866 cpu_manager.go:414] "containerMap: removing container" podUID="186b144a-c571-47a5-8ca0-264b72fcaea4" containerName="maas-api" Nov 02 14:34:16.415129 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.415129 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186b144a-c571-47a5-8ca0-264b72fcaea4" containerName="maas-api" Nov 02 14:34:16.415316 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.415302 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="186b144a-c571-47a5-8ca0-264b72fcaea4" containerName="maas-api" Nov 02 14:34:16.419125 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.419105 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw" Nov 02 14:34:16.438743 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.438719 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-778b2bce\"" Nov 02 14:34:16.445777 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.445748 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw"] Nov 02 14:34:16.467655 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.467625 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-778b2bce-dockercfg-w54p6\"" Nov 02 14:34:16.504264 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.504229 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh"] Nov 02 14:34:16.508844 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.508807 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh" Nov 02 14:34:16.533989 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.533149 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh"] Nov 02 14:34:16.570896 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.570725 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj5w2\" (UniqueName: \"kubernetes.io/projected/39897b77-0b2d-42b1-9c4f-77cfe4025791-kube-api-access-kj5w2\") pod \"payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw\" (UID: \"39897b77-0b2d-42b1-9c4f-77cfe4025791\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw" Nov 02 14:34:16.570896 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.570823 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39897b77-0b2d-42b1-9c4f-77cfe4025791-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw\" (UID: \"39897b77-0b2d-42b1-9c4f-77cfe4025791\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw" Nov 02 14:34:16.672335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.672246 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj5w2\" (UniqueName: \"kubernetes.io/projected/39897b77-0b2d-42b1-9c4f-77cfe4025791-kube-api-access-kj5w2\") pod \"payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw\" (UID: \"39897b77-0b2d-42b1-9c4f-77cfe4025791\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw" Nov 02 14:34:16.672335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.672330 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39897b77-0b2d-42b1-9c4f-77cfe4025791-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw\" (UID: \"39897b77-0b2d-42b1-9c4f-77cfe4025791\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw" Nov 02 14:34:16.672574 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.672382 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-585x5\" (UniqueName: \"kubernetes.io/projected/f9d55902-d242-4c33-98dd-0c273256c497-kube-api-access-585x5\") pod \"payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh\" (UID: \"f9d55902-d242-4c33-98dd-0c273256c497\") " pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh" Nov 02 14:34:16.672574 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.672534 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9d55902-d242-4c33-98dd-0c273256c497-plugins-config-volume\") pod \"payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh\" (UID: \"f9d55902-d242-4c33-98dd-0c273256c497\") " pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh" Nov 02 14:34:16.673270 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.673249 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39897b77-0b2d-42b1-9c4f-77cfe4025791-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw\" (UID: \"39897b77-0b2d-42b1-9c4f-77cfe4025791\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw" Nov 02 14:34:16.695963 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.695915 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj5w2\" (UniqueName: \"kubernetes.io/projected/39897b77-0b2d-42b1-9c4f-77cfe4025791-kube-api-access-kj5w2\") pod \"payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw\" (UID: \"39897b77-0b2d-42b1-9c4f-77cfe4025791\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw" Nov 02 14:34:16.732163 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.732130 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw" Nov 02 14:34:16.776260 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.776120 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9d55902-d242-4c33-98dd-0c273256c497-plugins-config-volume\") pod \"payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh\" (UID: \"f9d55902-d242-4c33-98dd-0c273256c497\") " pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh" Nov 02 14:34:16.776551 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.776519 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-585x5\" (UniqueName: \"kubernetes.io/projected/f9d55902-d242-4c33-98dd-0c273256c497-kube-api-access-585x5\") pod \"payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh\" (UID: \"f9d55902-d242-4c33-98dd-0c273256c497\") " pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh" Nov 02 14:34:16.777063 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.777044 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9d55902-d242-4c33-98dd-0c273256c497-plugins-config-volume\") pod \"payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh\" (UID: \"f9d55902-d242-4c33-98dd-0c273256c497\") " pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh" Nov 02 14:34:16.800779 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.800367 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-585x5\" (UniqueName: \"kubernetes.io/projected/f9d55902-d242-4c33-98dd-0c273256c497-kube-api-access-585x5\") pod \"payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh\" (UID: \"f9d55902-d242-4c33-98dd-0c273256c497\") " pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh" Nov 02 14:34:16.832813 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:16.832778 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh" Nov 02 14:34:17.074103 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:17.074074 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw"] Nov 02 14:34:17.075612 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:34:17.075586 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39897b77_0b2d_42b1_9c4f_77cfe4025791.slice/crio-6402b08a4f7ac2b2f8380efd78c7950d0b29f58b06f5201da4ec1da5aaeccbf3 WatchSource:0}: Error finding container 6402b08a4f7ac2b2f8380efd78c7950d0b29f58b06f5201da4ec1da5aaeccbf3: Status 404 returned error can't find the container with id 6402b08a4f7ac2b2f8380efd78c7950d0b29f58b06f5201da4ec1da5aaeccbf3 Nov 02 14:34:17.150592 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:17.150562 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh"] Nov 02 14:34:17.152011 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:34:17.151988 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d55902_d242_4c33_98dd_0c273256c497.slice/crio-e3dd817966571d76a3ed1e49563a5b43e4549fafe809e19c52af8bb08b821cf3 WatchSource:0}: Error finding container e3dd817966571d76a3ed1e49563a5b43e4549fafe809e19c52af8bb08b821cf3: Status 404 returned error can't find the container with id e3dd817966571d76a3ed1e49563a5b43e4549fafe809e19c52af8bb08b821cf3 Nov 02 14:34:17.312069 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:17.311989 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw" event={"ID":"39897b77-0b2d-42b1-9c4f-77cfe4025791","Type":"ContainerStarted","Data":"0e7247bb7005f460b1f5e2bce497c92a585412160348d7814ab694c3008e4dc2"} Nov 02 14:34:17.312069 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:17.312062 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw" event={"ID":"39897b77-0b2d-42b1-9c4f-77cfe4025791","Type":"ContainerStarted","Data":"6402b08a4f7ac2b2f8380efd78c7950d0b29f58b06f5201da4ec1da5aaeccbf3"} Nov 02 14:34:17.312305 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:17.312182 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw" Nov 02 14:34:17.313567 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:17.313539 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh" event={"ID":"f9d55902-d242-4c33-98dd-0c273256c497","Type":"ContainerStarted","Data":"2aa7165b7d5ea58e732f95c89159f6e19d7b7483c45c74783d4d307987149d15"} Nov 02 14:34:17.313677 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:17.313580 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh" event={"ID":"f9d55902-d242-4c33-98dd-0c273256c497","Type":"ContainerStarted","Data":"e3dd817966571d76a3ed1e49563a5b43e4549fafe809e19c52af8bb08b821cf3"} Nov 02 14:34:17.313677 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:17.313623 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh" Nov 02 14:34:17.354555 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:17.354132 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw" podStartSLOduration=1.354107103 podStartE2EDuration="1.354107103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:34:17.352874382 +0000 UTC m=+2222.536721719" watchObservedRunningTime="2026-11-02 14:34:17.354107103 +0000 UTC m=+2222.537954443" Nov 02 14:34:17.399445 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:17.399388 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh" podStartSLOduration=1.399373709 podStartE2EDuration="1.399373709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:34:17.398663574 +0000 UTC m=+2222.582510912" watchObservedRunningTime="2026-11-02 14:34:17.399373709 +0000 UTC m=+2222.583221045" Nov 02 14:34:28.323620 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:28.323584 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh" Nov 02 14:34:28.324155 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:28.323787 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw" Nov 02 14:34:36.011646 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:34:36.011607 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:34:40.657351 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:34:40.657296 4866 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" podUID="fcd82cc88301759f1e08e2b014a50651" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Nov 02 14:35:06.011254 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:35:06.011216 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:35:13.462469 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:13.462428 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf"] Nov 02 14:35:18.462817 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:18.462766 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" podUID="f06bd94b-7a9c-4463-8d81-1650b45cafdc" containerName="payload-processing" containerID="cri-o://af1ae0442c88c59475cb4dcd62b7aa56b25dcd590d3c803c17cee03085b3ad8f" gracePeriod=25 Nov 02 14:35:18.726552 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:18.726519 4866 generic.go:356] "Generic (PLEG): container finished" podID="f06bd94b-7a9c-4463-8d81-1650b45cafdc" containerID="af1ae0442c88c59475cb4dcd62b7aa56b25dcd590d3c803c17cee03085b3ad8f" exitCode=0 Nov 02 14:35:18.726691 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:18.726587 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" event={"ID":"f06bd94b-7a9c-4463-8d81-1650b45cafdc","Type":"ContainerDied","Data":"af1ae0442c88c59475cb4dcd62b7aa56b25dcd590d3c803c17cee03085b3ad8f"} Nov 02 14:35:18.726691 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:18.726644 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" event={"ID":"f06bd94b-7a9c-4463-8d81-1650b45cafdc","Type":"ContainerDied","Data":"72b0746114c13d9a95de6f40912784cbfd4fbdea532e85cec6f7060ef4042e5d"} Nov 02 14:35:18.726691 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:18.726663 4866 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72b0746114c13d9a95de6f40912784cbfd4fbdea532e85cec6f7060ef4042e5d" Nov 02 14:35:18.751579 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:18.751553 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" Nov 02 14:35:18.816577 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:18.816541 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f06bd94b-7a9c-4463-8d81-1650b45cafdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06bd94b-7a9c-4463-8d81-1650b45cafdc-plugins-config-volume\") pod \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\" (UID: \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\") " Nov 02 14:35:18.816788 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:18.816674 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f06bd94b-7a9c-4463-8d81-1650b45cafdc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f06bd94b-7a9c-4463-8d81-1650b45cafdc-routing-overlay-volume\") pod \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\" (UID: \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\") " Nov 02 14:35:18.816788 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:18.816748 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06bd94b-7a9c-4463-8d81-1650b45cafdc-kube-api-access-9lbsd\" (UniqueName: \"kubernetes.io/projected/f06bd94b-7a9c-4463-8d81-1650b45cafdc-kube-api-access-9lbsd\") pod \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\" (UID: \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\") " Nov 02 14:35:18.816930 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:18.816831 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06bd94b-7a9c-4463-8d81-1650b45cafdc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f06bd94b-7a9c-4463-8d81-1650b45cafdc-provider-credentials\") pod \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\" (UID: \"f06bd94b-7a9c-4463-8d81-1650b45cafdc\") " Nov 02 14:35:18.817115 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:18.817087 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06bd94b-7a9c-4463-8d81-1650b45cafdc-routing-overlay-volume" pod "f06bd94b-7a9c-4463-8d81-1650b45cafdc" (UID: "f06bd94b-7a9c-4463-8d81-1650b45cafdc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:35:18.817193 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:18.817137 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06bd94b-7a9c-4463-8d81-1650b45cafdc-plugins-config-volume" pod "f06bd94b-7a9c-4463-8d81-1650b45cafdc" (UID: "f06bd94b-7a9c-4463-8d81-1650b45cafdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:35:18.817493 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:18.817462 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06bd94b-7a9c-4463-8d81-1650b45cafdc-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:35:18.817564 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:18.817493 4866 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f06bd94b-7a9c-4463-8d81-1650b45cafdc-routing-overlay-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:35:18.819746 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:18.819711 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06bd94b-7a9c-4463-8d81-1650b45cafdc-provider-credentials" pod "f06bd94b-7a9c-4463-8d81-1650b45cafdc" (UID: "f06bd94b-7a9c-4463-8d81-1650b45cafdc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:35:18.819941 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:18.819919 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06bd94b-7a9c-4463-8d81-1650b45cafdc-kube-api-access-9lbsd" pod "f06bd94b-7a9c-4463-8d81-1650b45cafdc" (UID: "f06bd94b-7a9c-4463-8d81-1650b45cafdc"). InnerVolumeSpecName "kube-api-access-9lbsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:35:18.923957 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:18.919186 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lbsd\" (UniqueName: \"kubernetes.io/projected/f06bd94b-7a9c-4463-8d81-1650b45cafdc-kube-api-access-9lbsd\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:35:18.923957 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:18.919256 4866 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f06bd94b-7a9c-4463-8d81-1650b45cafdc-provider-credentials\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:35:19.733204 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:19.733161 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf" Nov 02 14:35:19.790747 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:19.790710 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf"] Nov 02 14:35:19.802695 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:19.802661 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-b55cg77kf"] Nov 02 14:35:20.947876 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:20.947838 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f06bd94b-7a9c-4463-8d81-1650b45cafdc" path="/var/lib/kubelet/pods/f06bd94b-7a9c-4463-8d81-1650b45cafdc/volumes" Nov 02 14:35:24.969782 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:24.969745 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg"] Nov 02 14:35:29.971121 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:29.971024 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" podUID="4991d22f-27fd-44d9-bcfe-2f991088c0bc" containerName="payload-processing" containerID="cri-o://fdc96af376b40c3e4eabe3d95ddc9e2d0ad09f114cd54e637e12f874f8c8fdd2" gracePeriod=25 Nov 02 14:35:30.252661 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.252632 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" Nov 02 14:35:30.372555 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.372513 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4991d22f-27fd-44d9-bcfe-2f991088c0bc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4991d22f-27fd-44d9-bcfe-2f991088c0bc-provider-credentials\") pod \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\" (UID: \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\") " Nov 02 14:35:30.372733 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.372591 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4991d22f-27fd-44d9-bcfe-2f991088c0bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4991d22f-27fd-44d9-bcfe-2f991088c0bc-plugins-config-volume\") pod \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\" (UID: \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\") " Nov 02 14:35:30.372733 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.372637 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4991d22f-27fd-44d9-bcfe-2f991088c0bc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4991d22f-27fd-44d9-bcfe-2f991088c0bc-routing-overlay-volume\") pod \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\" (UID: \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\") " Nov 02 14:35:30.372854 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.372834 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4991d22f-27fd-44d9-bcfe-2f991088c0bc-kube-api-access-stch8\" (UniqueName: \"kubernetes.io/projected/4991d22f-27fd-44d9-bcfe-2f991088c0bc-kube-api-access-stch8\") pod \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\" (UID: \"4991d22f-27fd-44d9-bcfe-2f991088c0bc\") " Nov 02 14:35:30.373109 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.373080 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4991d22f-27fd-44d9-bcfe-2f991088c0bc-routing-overlay-volume" pod "4991d22f-27fd-44d9-bcfe-2f991088c0bc" (UID: "4991d22f-27fd-44d9-bcfe-2f991088c0bc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:35:30.373109 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.373096 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4991d22f-27fd-44d9-bcfe-2f991088c0bc-plugins-config-volume" pod "4991d22f-27fd-44d9-bcfe-2f991088c0bc" (UID: "4991d22f-27fd-44d9-bcfe-2f991088c0bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:35:30.373425 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.373400 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4991d22f-27fd-44d9-bcfe-2f991088c0bc-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:35:30.373546 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.373429 4866 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4991d22f-27fd-44d9-bcfe-2f991088c0bc-routing-overlay-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:35:30.374723 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.374703 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4991d22f-27fd-44d9-bcfe-2f991088c0bc-provider-credentials" pod "4991d22f-27fd-44d9-bcfe-2f991088c0bc" (UID: "4991d22f-27fd-44d9-bcfe-2f991088c0bc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:35:30.374937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.374909 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4991d22f-27fd-44d9-bcfe-2f991088c0bc-kube-api-access-stch8" pod "4991d22f-27fd-44d9-bcfe-2f991088c0bc" (UID: "4991d22f-27fd-44d9-bcfe-2f991088c0bc"). InnerVolumeSpecName "kube-api-access-stch8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:35:30.474536 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.474433 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stch8\" (UniqueName: \"kubernetes.io/projected/4991d22f-27fd-44d9-bcfe-2f991088c0bc-kube-api-access-stch8\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:35:30.474536 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.474471 4866 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4991d22f-27fd-44d9-bcfe-2f991088c0bc-provider-credentials\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:35:30.803563 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.803464 4866 generic.go:356] "Generic (PLEG): container finished" podID="4991d22f-27fd-44d9-bcfe-2f991088c0bc" containerID="fdc96af376b40c3e4eabe3d95ddc9e2d0ad09f114cd54e637e12f874f8c8fdd2" exitCode=0 Nov 02 14:35:30.803563 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.803514 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" Nov 02 14:35:30.803847 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.803550 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" event={"ID":"4991d22f-27fd-44d9-bcfe-2f991088c0bc","Type":"ContainerDied","Data":"fdc96af376b40c3e4eabe3d95ddc9e2d0ad09f114cd54e637e12f874f8c8fdd2"} Nov 02 14:35:30.803847 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.803650 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg" event={"ID":"4991d22f-27fd-44d9-bcfe-2f991088c0bc","Type":"ContainerDied","Data":"bb0b3c195a7b22e59b9140067ebf53aea188d1ddd1b4ab588f6d447aa207642f"} Nov 02 14:35:30.803847 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.803681 4866 scope.go:122] "RemoveContainer" containerID="fdc96af376b40c3e4eabe3d95ddc9e2d0ad09f114cd54e637e12f874f8c8fdd2" Nov 02 14:35:30.814541 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.814522 4866 scope.go:122] "RemoveContainer" containerID="fdc96af376b40c3e4eabe3d95ddc9e2d0ad09f114cd54e637e12f874f8c8fdd2" Nov 02 14:35:30.814812 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:35:30.814790 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc96af376b40c3e4eabe3d95ddc9e2d0ad09f114cd54e637e12f874f8c8fdd2\": container with ID starting with fdc96af376b40c3e4eabe3d95ddc9e2d0ad09f114cd54e637e12f874f8c8fdd2 not found: ID does not exist" containerID="fdc96af376b40c3e4eabe3d95ddc9e2d0ad09f114cd54e637e12f874f8c8fdd2" Nov 02 14:35:30.814858 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.814821 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc96af376b40c3e4eabe3d95ddc9e2d0ad09f114cd54e637e12f874f8c8fdd2"} err="failed to get container status \"fdc96af376b40c3e4eabe3d95ddc9e2d0ad09f114cd54e637e12f874f8c8fdd2\": rpc error: code = NotFound desc = could not find container \"fdc96af376b40c3e4eabe3d95ddc9e2d0ad09f114cd54e637e12f874f8c8fdd2\": container with ID starting with fdc96af376b40c3e4eabe3d95ddc9e2d0ad09f114cd54e637e12f874f8c8fdd2 not found: ID does not exist" Nov 02 14:35:30.856583 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.856535 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg"] Nov 02 14:35:30.868077 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.868043 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-89bdmjgwg"] Nov 02 14:35:30.947361 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:30.947325 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4991d22f-27fd-44d9-bcfe-2f991088c0bc" path="/var/lib/kubelet/pods/4991d22f-27fd-44d9-bcfe-2f991088c0bc/volumes" Nov 02 14:35:36.010521 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:35:36.010484 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:35:55.189294 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:55.189259 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m"] Nov 02 14:35:55.224907 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:35:55.224871 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq"] Nov 02 14:36:00.189661 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:00.189615 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m" podUID="d73bd157-9369-406e-b2bf-d4cfa38a1d33" containerName="payload-processing" containerID="cri-o://401c5fae59a8743d91485251ffa133226e1d2a261fcbb5b6d6b3975c14a18a00" gracePeriod=25 Nov 02 14:36:00.225783 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:00.225742 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq" podUID="ae683718-049c-4ea8-82da-4f97e31175ac" containerName="payload-pre-processing" containerID="cri-o://f2e3b6d90690af160aca9edf213c0c0a4a34e8345344ddb6dc8880a84536b0f6" gracePeriod=25 Nov 02 14:36:00.534811 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:00.534784 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m" Nov 02 14:36:00.563169 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:00.563142 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq" Nov 02 14:36:00.576910 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:00.576866 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73bd157-9369-406e-b2bf-d4cfa38a1d33-kube-api-access-wkd4r\" (UniqueName: \"kubernetes.io/projected/d73bd157-9369-406e-b2bf-d4cfa38a1d33-kube-api-access-wkd4r\") pod \"d73bd157-9369-406e-b2bf-d4cfa38a1d33\" (UID: \"d73bd157-9369-406e-b2bf-d4cfa38a1d33\") " Nov 02 14:36:00.577224 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:00.577197 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d73bd157-9369-406e-b2bf-d4cfa38a1d33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73bd157-9369-406e-b2bf-d4cfa38a1d33-plugins-config-volume\") pod \"d73bd157-9369-406e-b2bf-d4cfa38a1d33\" (UID: \"d73bd157-9369-406e-b2bf-d4cfa38a1d33\") " Nov 02 14:36:00.577597 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:00.577573 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73bd157-9369-406e-b2bf-d4cfa38a1d33-plugins-config-volume" pod "d73bd157-9369-406e-b2bf-d4cfa38a1d33" (UID: "d73bd157-9369-406e-b2bf-d4cfa38a1d33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:00.577739 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:00.577722 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73bd157-9369-406e-b2bf-d4cfa38a1d33-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:00.580283 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:00.580249 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73bd157-9369-406e-b2bf-d4cfa38a1d33-kube-api-access-wkd4r" pod "d73bd157-9369-406e-b2bf-d4cfa38a1d33" (UID: "d73bd157-9369-406e-b2bf-d4cfa38a1d33"). InnerVolumeSpecName "kube-api-access-wkd4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:00.678978 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:00.678916 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae683718-049c-4ea8-82da-4f97e31175ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae683718-049c-4ea8-82da-4f97e31175ac-plugins-config-volume\") pod \"ae683718-049c-4ea8-82da-4f97e31175ac\" (UID: \"ae683718-049c-4ea8-82da-4f97e31175ac\") " Nov 02 14:36:00.679171 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:00.679100 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae683718-049c-4ea8-82da-4f97e31175ac-kube-api-access-dz9zl\" (UniqueName: \"kubernetes.io/projected/ae683718-049c-4ea8-82da-4f97e31175ac-kube-api-access-dz9zl\") pod \"ae683718-049c-4ea8-82da-4f97e31175ac\" (UID: \"ae683718-049c-4ea8-82da-4f97e31175ac\") " Nov 02 14:36:00.679352 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:00.679327 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae683718-049c-4ea8-82da-4f97e31175ac-plugins-config-volume" pod "ae683718-049c-4ea8-82da-4f97e31175ac" (UID: "ae683718-049c-4ea8-82da-4f97e31175ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:00.679625 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:00.679605 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkd4r\" (UniqueName: \"kubernetes.io/projected/d73bd157-9369-406e-b2bf-d4cfa38a1d33-kube-api-access-wkd4r\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:00.679707 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:00.679629 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae683718-049c-4ea8-82da-4f97e31175ac-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:00.681876 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:00.681844 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae683718-049c-4ea8-82da-4f97e31175ac-kube-api-access-dz9zl" pod "ae683718-049c-4ea8-82da-4f97e31175ac" (UID: "ae683718-049c-4ea8-82da-4f97e31175ac"). InnerVolumeSpecName "kube-api-access-dz9zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:00.780687 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:00.780603 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz9zl\" (UniqueName: \"kubernetes.io/projected/ae683718-049c-4ea8-82da-4f97e31175ac-kube-api-access-dz9zl\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:01.000493 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:01.000462 4866 generic.go:356] "Generic (PLEG): container finished" podID="ae683718-049c-4ea8-82da-4f97e31175ac" containerID="f2e3b6d90690af160aca9edf213c0c0a4a34e8345344ddb6dc8880a84536b0f6" exitCode=0 Nov 02 14:36:01.000666 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:01.000503 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq" Nov 02 14:36:01.000666 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:01.000544 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq" event={"ID":"ae683718-049c-4ea8-82da-4f97e31175ac","Type":"ContainerDied","Data":"f2e3b6d90690af160aca9edf213c0c0a4a34e8345344ddb6dc8880a84536b0f6"} Nov 02 14:36:01.000666 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:01.000592 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq" event={"ID":"ae683718-049c-4ea8-82da-4f97e31175ac","Type":"ContainerDied","Data":"3872eac538780f89c0dfb827ba95323dbb8175ad9bc121ed4e551a7f0419bffe"} Nov 02 14:36:01.000666 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:01.000620 4866 scope.go:122] "RemoveContainer" containerID="f2e3b6d90690af160aca9edf213c0c0a4a34e8345344ddb6dc8880a84536b0f6" Nov 02 14:36:01.002221 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:01.002197 4866 generic.go:356] "Generic (PLEG): container finished" podID="d73bd157-9369-406e-b2bf-d4cfa38a1d33" containerID="401c5fae59a8743d91485251ffa133226e1d2a261fcbb5b6d6b3975c14a18a00" exitCode=0 Nov 02 14:36:01.002344 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:01.002237 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m" event={"ID":"d73bd157-9369-406e-b2bf-d4cfa38a1d33","Type":"ContainerDied","Data":"401c5fae59a8743d91485251ffa133226e1d2a261fcbb5b6d6b3975c14a18a00"} Nov 02 14:36:01.002344 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:01.002277 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m" event={"ID":"d73bd157-9369-406e-b2bf-d4cfa38a1d33","Type":"ContainerDied","Data":"d52aa3eeaa4dec3fad224e4d7205eb9cd563642c41622fec5e732b9fa6a862a3"} Nov 02 14:36:01.002344 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:01.002243 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m" Nov 02 14:36:01.010723 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:01.010702 4866 scope.go:122] "RemoveContainer" containerID="f2e3b6d90690af160aca9edf213c0c0a4a34e8345344ddb6dc8880a84536b0f6" Nov 02 14:36:01.011018 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:36:01.010993 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e3b6d90690af160aca9edf213c0c0a4a34e8345344ddb6dc8880a84536b0f6\": container with ID starting with f2e3b6d90690af160aca9edf213c0c0a4a34e8345344ddb6dc8880a84536b0f6 not found: ID does not exist" containerID="f2e3b6d90690af160aca9edf213c0c0a4a34e8345344ddb6dc8880a84536b0f6" Nov 02 14:36:01.011089 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:01.011024 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e3b6d90690af160aca9edf213c0c0a4a34e8345344ddb6dc8880a84536b0f6"} err="failed to get container status \"f2e3b6d90690af160aca9edf213c0c0a4a34e8345344ddb6dc8880a84536b0f6\": rpc error: code = NotFound desc = could not find container \"f2e3b6d90690af160aca9edf213c0c0a4a34e8345344ddb6dc8880a84536b0f6\": container with ID starting with f2e3b6d90690af160aca9edf213c0c0a4a34e8345344ddb6dc8880a84536b0f6 not found: ID does not exist" Nov 02 14:36:01.011089 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:01.011041 4866 scope.go:122] "RemoveContainer" containerID="401c5fae59a8743d91485251ffa133226e1d2a261fcbb5b6d6b3975c14a18a00" Nov 02 14:36:01.019789 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:01.019757 4866 scope.go:122] "RemoveContainer" containerID="401c5fae59a8743d91485251ffa133226e1d2a261fcbb5b6d6b3975c14a18a00" Nov 02 14:36:01.020106 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:36:01.020085 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"401c5fae59a8743d91485251ffa133226e1d2a261fcbb5b6d6b3975c14a18a00\": container with ID starting with 401c5fae59a8743d91485251ffa133226e1d2a261fcbb5b6d6b3975c14a18a00 not found: ID does not exist" containerID="401c5fae59a8743d91485251ffa133226e1d2a261fcbb5b6d6b3975c14a18a00" Nov 02 14:36:01.020181 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:01.020110 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"401c5fae59a8743d91485251ffa133226e1d2a261fcbb5b6d6b3975c14a18a00"} err="failed to get container status \"401c5fae59a8743d91485251ffa133226e1d2a261fcbb5b6d6b3975c14a18a00\": rpc error: code = NotFound desc = could not find container \"401c5fae59a8743d91485251ffa133226e1d2a261fcbb5b6d6b3975c14a18a00\": container with ID starting with 401c5fae59a8743d91485251ffa133226e1d2a261fcbb5b6d6b3975c14a18a00 not found: ID does not exist" Nov 02 14:36:01.055907 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:01.055816 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq"] Nov 02 14:36:01.066214 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:01.066171 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-90a96161-86597f9cbc-vc9cq"] Nov 02 14:36:01.094518 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:01.094486 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m"] Nov 02 14:36:01.100821 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:01.100792 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-90a96161-694f45b47c-t985m"] Nov 02 14:36:02.947266 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:02.947232 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae683718-049c-4ea8-82da-4f97e31175ac" path="/var/lib/kubelet/pods/ae683718-049c-4ea8-82da-4f97e31175ac/volumes" Nov 02 14:36:02.947811 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:02.947795 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d73bd157-9369-406e-b2bf-d4cfa38a1d33" path="/var/lib/kubelet/pods/d73bd157-9369-406e-b2bf-d4cfa38a1d33/volumes" Nov 02 14:36:05.208322 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:05.208289 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh"] Nov 02 14:36:05.241811 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:05.241772 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw"] Nov 02 14:36:06.011280 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:36:06.011242 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:36:10.209555 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:10.209509 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh" podUID="f9d55902-d242-4c33-98dd-0c273256c497" containerName="payload-processing" containerID="cri-o://2aa7165b7d5ea58e732f95c89159f6e19d7b7483c45c74783d4d307987149d15" gracePeriod=25 Nov 02 14:36:10.242894 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:10.242852 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw" podUID="39897b77-0b2d-42b1-9c4f-77cfe4025791" containerName="payload-pre-processing" containerID="cri-o://0e7247bb7005f460b1f5e2bce497c92a585412160348d7814ab694c3008e4dc2" gracePeriod=25 Nov 02 14:36:10.543454 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:10.543180 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh" Nov 02 14:36:10.579604 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:10.579575 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw" Nov 02 14:36:10.731045 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:10.730930 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39897b77-0b2d-42b1-9c4f-77cfe4025791-kube-api-access-kj5w2\" (UniqueName: \"kubernetes.io/projected/39897b77-0b2d-42b1-9c4f-77cfe4025791-kube-api-access-kj5w2\") pod \"39897b77-0b2d-42b1-9c4f-77cfe4025791\" (UID: \"39897b77-0b2d-42b1-9c4f-77cfe4025791\") " Nov 02 14:36:10.731045 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:10.731007 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d55902-d242-4c33-98dd-0c273256c497-kube-api-access-585x5\" (UniqueName: \"kubernetes.io/projected/f9d55902-d242-4c33-98dd-0c273256c497-kube-api-access-585x5\") pod \"f9d55902-d242-4c33-98dd-0c273256c497\" (UID: \"f9d55902-d242-4c33-98dd-0c273256c497\") " Nov 02 14:36:10.731243 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:10.731082 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9d55902-d242-4c33-98dd-0c273256c497-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9d55902-d242-4c33-98dd-0c273256c497-plugins-config-volume\") pod \"f9d55902-d242-4c33-98dd-0c273256c497\" (UID: \"f9d55902-d242-4c33-98dd-0c273256c497\") " Nov 02 14:36:10.731243 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:10.731209 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39897b77-0b2d-42b1-9c4f-77cfe4025791-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39897b77-0b2d-42b1-9c4f-77cfe4025791-plugins-config-volume\") pod \"39897b77-0b2d-42b1-9c4f-77cfe4025791\" (UID: \"39897b77-0b2d-42b1-9c4f-77cfe4025791\") " Nov 02 14:36:10.731521 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:10.731495 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d55902-d242-4c33-98dd-0c273256c497-plugins-config-volume" pod "f9d55902-d242-4c33-98dd-0c273256c497" (UID: "f9d55902-d242-4c33-98dd-0c273256c497"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:10.731608 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:10.731548 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39897b77-0b2d-42b1-9c4f-77cfe4025791-plugins-config-volume" pod "39897b77-0b2d-42b1-9c4f-77cfe4025791" (UID: "39897b77-0b2d-42b1-9c4f-77cfe4025791"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:10.731750 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:10.731734 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39897b77-0b2d-42b1-9c4f-77cfe4025791-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:10.731809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:10.731756 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9d55902-d242-4c33-98dd-0c273256c497-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:10.733045 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:10.733020 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39897b77-0b2d-42b1-9c4f-77cfe4025791-kube-api-access-kj5w2" pod "39897b77-0b2d-42b1-9c4f-77cfe4025791" (UID: "39897b77-0b2d-42b1-9c4f-77cfe4025791"). InnerVolumeSpecName "kube-api-access-kj5w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:10.733148 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:10.733115 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d55902-d242-4c33-98dd-0c273256c497-kube-api-access-585x5" pod "f9d55902-d242-4c33-98dd-0c273256c497" (UID: "f9d55902-d242-4c33-98dd-0c273256c497"). InnerVolumeSpecName "kube-api-access-585x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:10.833295 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:10.833255 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj5w2\" (UniqueName: \"kubernetes.io/projected/39897b77-0b2d-42b1-9c4f-77cfe4025791-kube-api-access-kj5w2\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:10.833295 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:10.833291 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-585x5\" (UniqueName: \"kubernetes.io/projected/f9d55902-d242-4c33-98dd-0c273256c497-kube-api-access-585x5\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:11.083436 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:11.083346 4866 generic.go:356] "Generic (PLEG): container finished" podID="f9d55902-d242-4c33-98dd-0c273256c497" containerID="2aa7165b7d5ea58e732f95c89159f6e19d7b7483c45c74783d4d307987149d15" exitCode=0 Nov 02 14:36:11.083436 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:11.083402 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh" Nov 02 14:36:11.083667 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:11.083440 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh" event={"ID":"f9d55902-d242-4c33-98dd-0c273256c497","Type":"ContainerDied","Data":"2aa7165b7d5ea58e732f95c89159f6e19d7b7483c45c74783d4d307987149d15"} Nov 02 14:36:11.083667 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:11.083522 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh" event={"ID":"f9d55902-d242-4c33-98dd-0c273256c497","Type":"ContainerDied","Data":"e3dd817966571d76a3ed1e49563a5b43e4549fafe809e19c52af8bb08b821cf3"} Nov 02 14:36:11.083667 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:11.083551 4866 scope.go:122] "RemoveContainer" containerID="2aa7165b7d5ea58e732f95c89159f6e19d7b7483c45c74783d4d307987149d15" Nov 02 14:36:11.085088 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:11.085067 4866 generic.go:356] "Generic (PLEG): container finished" podID="39897b77-0b2d-42b1-9c4f-77cfe4025791" containerID="0e7247bb7005f460b1f5e2bce497c92a585412160348d7814ab694c3008e4dc2" exitCode=0 Nov 02 14:36:11.085208 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:11.085102 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw" event={"ID":"39897b77-0b2d-42b1-9c4f-77cfe4025791","Type":"ContainerDied","Data":"0e7247bb7005f460b1f5e2bce497c92a585412160348d7814ab694c3008e4dc2"} Nov 02 14:36:11.085208 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:11.085134 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw" Nov 02 14:36:11.085326 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:11.085137 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw" event={"ID":"39897b77-0b2d-42b1-9c4f-77cfe4025791","Type":"ContainerDied","Data":"6402b08a4f7ac2b2f8380efd78c7950d0b29f58b06f5201da4ec1da5aaeccbf3"} Nov 02 14:36:11.097522 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:11.097496 4866 scope.go:122] "RemoveContainer" containerID="2aa7165b7d5ea58e732f95c89159f6e19d7b7483c45c74783d4d307987149d15" Nov 02 14:36:11.098009 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:36:11.097983 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aa7165b7d5ea58e732f95c89159f6e19d7b7483c45c74783d4d307987149d15\": container with ID starting with 2aa7165b7d5ea58e732f95c89159f6e19d7b7483c45c74783d4d307987149d15 not found: ID does not exist" containerID="2aa7165b7d5ea58e732f95c89159f6e19d7b7483c45c74783d4d307987149d15" Nov 02 14:36:11.098106 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:11.098020 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aa7165b7d5ea58e732f95c89159f6e19d7b7483c45c74783d4d307987149d15"} err="failed to get container status \"2aa7165b7d5ea58e732f95c89159f6e19d7b7483c45c74783d4d307987149d15\": rpc error: code = NotFound desc = could not find container \"2aa7165b7d5ea58e732f95c89159f6e19d7b7483c45c74783d4d307987149d15\": container with ID starting with 2aa7165b7d5ea58e732f95c89159f6e19d7b7483c45c74783d4d307987149d15 not found: ID does not exist" Nov 02 14:36:11.098106 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:11.098048 4866 scope.go:122] "RemoveContainer" containerID="0e7247bb7005f460b1f5e2bce497c92a585412160348d7814ab694c3008e4dc2" Nov 02 14:36:11.107242 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:11.107222 4866 scope.go:122] "RemoveContainer" containerID="0e7247bb7005f460b1f5e2bce497c92a585412160348d7814ab694c3008e4dc2" Nov 02 14:36:11.107533 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:36:11.107515 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e7247bb7005f460b1f5e2bce497c92a585412160348d7814ab694c3008e4dc2\": container with ID starting with 0e7247bb7005f460b1f5e2bce497c92a585412160348d7814ab694c3008e4dc2 not found: ID does not exist" containerID="0e7247bb7005f460b1f5e2bce497c92a585412160348d7814ab694c3008e4dc2" Nov 02 14:36:11.107577 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:11.107541 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e7247bb7005f460b1f5e2bce497c92a585412160348d7814ab694c3008e4dc2"} err="failed to get container status \"0e7247bb7005f460b1f5e2bce497c92a585412160348d7814ab694c3008e4dc2\": rpc error: code = NotFound desc = could not find container \"0e7247bb7005f460b1f5e2bce497c92a585412160348d7814ab694c3008e4dc2\": container with ID starting with 0e7247bb7005f460b1f5e2bce497c92a585412160348d7814ab694c3008e4dc2 not found: ID does not exist" Nov 02 14:36:11.135688 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:11.135653 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh"] Nov 02 14:36:11.140408 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:11.140382 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778b2bce-7f6cdb88c9-vbslh"] Nov 02 14:36:11.175771 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:11.175742 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw"] Nov 02 14:36:11.179580 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:11.179552 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778b2bce-5645b6b76c-595vw"] Nov 02 14:36:12.946042 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:12.946006 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39897b77-0b2d-42b1-9c4f-77cfe4025791" path="/var/lib/kubelet/pods/39897b77-0b2d-42b1-9c4f-77cfe4025791/volumes" Nov 02 14:36:12.946558 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:12.946542 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d55902-d242-4c33-98dd-0c273256c497" path="/var/lib/kubelet/pods/f9d55902-d242-4c33-98dd-0c273256c497/volumes" Nov 02 14:36:36.010906 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:36:36.010861 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:36:36.283096 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:36.283004 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n"] Nov 02 14:36:36.312407 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:36.312371 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs"] Nov 02 14:36:51.304242 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.304192 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" podUID="4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e" containerName="main" containerID="cri-o://ebd19e9adf5c442dc1691c9810a4fcd70d177c5fe7d1b3b6c3cec3262b7dc1e3" gracePeriod=45 Nov 02 14:36:51.332916 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.332877 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" podUID="0129df2e-9d21-45f7-9aa2-53777ce753b4" containerName="main" containerID="cri-o://c63776dc143f1026221371d412fff10fbd92460fdc8db061db38b4a44f3f1f58" gracePeriod=45 Nov 02 14:36:51.639712 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.639683 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:36:51.642754 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.642729 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:36:51.721913 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.721874 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-dshm\") pod \"0129df2e-9d21-45f7-9aa2-53777ce753b4\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " Nov 02 14:36:51.722129 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.721932 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-home\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-home\") pod \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " Nov 02 14:36:51.722129 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.722034 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0129df2e-9d21-45f7-9aa2-53777ce753b4-kube-api-access-2g48w\" (UniqueName: \"kubernetes.io/projected/0129df2e-9d21-45f7-9aa2-53777ce753b4-kube-api-access-2g48w\") pod \"0129df2e-9d21-45f7-9aa2-53777ce753b4\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " Nov 02 14:36:51.722129 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.722081 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-tmp-dir\") pod \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " Nov 02 14:36:51.722129 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.722123 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-tls-certs\") pod \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " Nov 02 14:36:51.722382 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.722199 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-tmp-dir\") pod \"0129df2e-9d21-45f7-9aa2-53777ce753b4\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " Nov 02 14:36:51.722382 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.722227 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-home" pod "4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e" (UID: "4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:51.722382 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.722280 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-home\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-home\") pod \"0129df2e-9d21-45f7-9aa2-53777ce753b4\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " Nov 02 14:36:51.722382 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.722307 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-dshm\") pod \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " Nov 02 14:36:51.722382 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.722344 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-model-cache\") pod \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " Nov 02 14:36:51.722692 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.722389 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-model-cache\") pod \"0129df2e-9d21-45f7-9aa2-53777ce753b4\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " Nov 02 14:36:51.722692 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.722435 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-tmp-dir" pod "4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e" (UID: "4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:51.722692 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.722451 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0129df2e-9d21-45f7-9aa2-53777ce753b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/0129df2e-9d21-45f7-9aa2-53777ce753b4-tls-certs\") pod \"0129df2e-9d21-45f7-9aa2-53777ce753b4\" (UID: \"0129df2e-9d21-45f7-9aa2-53777ce753b4\") " Nov 02 14:36:51.722692 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.722485 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-kube-api-access-r7q4h\" (UniqueName: \"kubernetes.io/projected/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-kube-api-access-r7q4h\") pod \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\" (UID: \"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e\") " Nov 02 14:36:51.722692 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.722517 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-tmp-dir" pod "0129df2e-9d21-45f7-9aa2-53777ce753b4" (UID: "0129df2e-9d21-45f7-9aa2-53777ce753b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:51.722971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.722707 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-home" pod "0129df2e-9d21-45f7-9aa2-53777ce753b4" (UID: "0129df2e-9d21-45f7-9aa2-53777ce753b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:51.722971 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.722830 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-model-cache" pod "0129df2e-9d21-45f7-9aa2-53777ce753b4" (UID: "0129df2e-9d21-45f7-9aa2-53777ce753b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:51.723276 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.723109 4866 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-home\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.723276 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.723134 4866 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-tmp-dir\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.723276 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.723154 4866 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-tmp-dir\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.723276 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.723169 4866 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-home\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.723276 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.723182 4866 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-model-cache\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.723276 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.723247 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-model-cache" pod "4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e" (UID: "4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:51.725203 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.725177 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-kube-api-access-r7q4h" pod "4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e" (UID: "4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e"). InnerVolumeSpecName "kube-api-access-r7q4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:51.725448 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.725421 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0129df2e-9d21-45f7-9aa2-53777ce753b4-tls-certs" pod "0129df2e-9d21-45f7-9aa2-53777ce753b4" (UID: "0129df2e-9d21-45f7-9aa2-53777ce753b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:36:51.726310 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.726286 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-dshm" pod "4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e" (UID: "4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:51.726310 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.726297 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-dshm" pod "0129df2e-9d21-45f7-9aa2-53777ce753b4" (UID: "0129df2e-9d21-45f7-9aa2-53777ce753b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:51.726566 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.726546 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0129df2e-9d21-45f7-9aa2-53777ce753b4-kube-api-access-2g48w" pod "0129df2e-9d21-45f7-9aa2-53777ce753b4" (UID: "0129df2e-9d21-45f7-9aa2-53777ce753b4"). InnerVolumeSpecName "kube-api-access-2g48w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:51.727456 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.727432 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-tls-certs" pod "4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e" (UID: "4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:36:51.824310 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.824275 4866 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0129df2e-9d21-45f7-9aa2-53777ce753b4-tls-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.824310 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.824305 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7q4h\" (UniqueName: \"kubernetes.io/projected/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-kube-api-access-r7q4h\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.824310 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.824314 4866 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0129df2e-9d21-45f7-9aa2-53777ce753b4-dshm\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.824545 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.824323 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g48w\" (UniqueName: \"kubernetes.io/projected/0129df2e-9d21-45f7-9aa2-53777ce753b4-kube-api-access-2g48w\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.824545 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.824334 4866 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-tls-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.824545 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.824342 4866 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-dshm\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.824545 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:51.824350 4866 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e-model-cache\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:52.349506 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.349468 4866 generic.go:356] "Generic (PLEG): container finished" podID="4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e" containerID="ebd19e9adf5c442dc1691c9810a4fcd70d177c5fe7d1b3b6c3cec3262b7dc1e3" exitCode=0 Nov 02 14:36:52.350046 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.349529 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" Nov 02 14:36:52.350046 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.349557 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" event={"ID":"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e","Type":"ContainerDied","Data":"ebd19e9adf5c442dc1691c9810a4fcd70d177c5fe7d1b3b6c3cec3262b7dc1e3"} Nov 02 14:36:52.350046 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.349594 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n" event={"ID":"4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e","Type":"ContainerDied","Data":"e4e81592ab000ac14610cddd5b97e81e4df524df4705bf4abeea0caded973c4f"} Nov 02 14:36:52.350046 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.349621 4866 scope.go:122] "RemoveContainer" containerID="ebd19e9adf5c442dc1691c9810a4fcd70d177c5fe7d1b3b6c3cec3262b7dc1e3" Nov 02 14:36:52.351171 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.351148 4866 generic.go:356] "Generic (PLEG): container finished" podID="0129df2e-9d21-45f7-9aa2-53777ce753b4" containerID="c63776dc143f1026221371d412fff10fbd92460fdc8db061db38b4a44f3f1f58" exitCode=0 Nov 02 14:36:52.351299 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.351183 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" event={"ID":"0129df2e-9d21-45f7-9aa2-53777ce753b4","Type":"ContainerDied","Data":"c63776dc143f1026221371d412fff10fbd92460fdc8db061db38b4a44f3f1f58"} Nov 02 14:36:52.351299 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.351239 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" event={"ID":"0129df2e-9d21-45f7-9aa2-53777ce753b4","Type":"ContainerDied","Data":"837aa1304f6fefb518f6a73a72829791fa66c4967a72747060f5be379db5f231"} Nov 02 14:36:52.351299 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.351199 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs" Nov 02 14:36:52.361828 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.361806 4866 scope.go:122] "RemoveContainer" containerID="ebd19e9adf5c442dc1691c9810a4fcd70d177c5fe7d1b3b6c3cec3262b7dc1e3" Nov 02 14:36:52.362268 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:36:52.362146 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebd19e9adf5c442dc1691c9810a4fcd70d177c5fe7d1b3b6c3cec3262b7dc1e3\": container with ID starting with ebd19e9adf5c442dc1691c9810a4fcd70d177c5fe7d1b3b6c3cec3262b7dc1e3 not found: ID does not exist" containerID="ebd19e9adf5c442dc1691c9810a4fcd70d177c5fe7d1b3b6c3cec3262b7dc1e3" Nov 02 14:36:52.362268 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.362172 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd19e9adf5c442dc1691c9810a4fcd70d177c5fe7d1b3b6c3cec3262b7dc1e3"} err="failed to get container status \"ebd19e9adf5c442dc1691c9810a4fcd70d177c5fe7d1b3b6c3cec3262b7dc1e3\": rpc error: code = NotFound desc = could not find container \"ebd19e9adf5c442dc1691c9810a4fcd70d177c5fe7d1b3b6c3cec3262b7dc1e3\": container with ID starting with ebd19e9adf5c442dc1691c9810a4fcd70d177c5fe7d1b3b6c3cec3262b7dc1e3 not found: ID does not exist" Nov 02 14:36:52.362268 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.362198 4866 scope.go:122] "RemoveContainer" containerID="c63776dc143f1026221371d412fff10fbd92460fdc8db061db38b4a44f3f1f58" Nov 02 14:36:52.370781 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.370763 4866 scope.go:122] "RemoveContainer" containerID="c63776dc143f1026221371d412fff10fbd92460fdc8db061db38b4a44f3f1f58" Nov 02 14:36:52.371061 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:36:52.371039 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c63776dc143f1026221371d412fff10fbd92460fdc8db061db38b4a44f3f1f58\": container with ID starting with c63776dc143f1026221371d412fff10fbd92460fdc8db061db38b4a44f3f1f58 not found: ID does not exist" containerID="c63776dc143f1026221371d412fff10fbd92460fdc8db061db38b4a44f3f1f58" Nov 02 14:36:52.371116 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.371069 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c63776dc143f1026221371d412fff10fbd92460fdc8db061db38b4a44f3f1f58"} err="failed to get container status \"c63776dc143f1026221371d412fff10fbd92460fdc8db061db38b4a44f3f1f58\": rpc error: code = NotFound desc = could not find container \"c63776dc143f1026221371d412fff10fbd92460fdc8db061db38b4a44f3f1f58\": container with ID starting with c63776dc143f1026221371d412fff10fbd92460fdc8db061db38b4a44f3f1f58 not found: ID does not exist" Nov 02 14:36:52.406392 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.406360 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n"] Nov 02 14:36:52.417930 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.417905 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/distinct-2-w2-4be645-kserve-66df89565d-xl64n"] Nov 02 14:36:52.452568 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.452536 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs"] Nov 02 14:36:52.459179 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.459152 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/simulator-w2-4be645-kserve-6c7984cfc9-p4qxs"] Nov 02 14:36:52.945703 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.945670 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0129df2e-9d21-45f7-9aa2-53777ce753b4" path="/var/lib/kubelet/pods/0129df2e-9d21-45f7-9aa2-53777ce753b4/volumes" Nov 02 14:36:52.946151 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:52.946137 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e" path="/var/lib/kubelet/pods/4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e/volumes" Nov 02 14:36:55.520479 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:55.520437 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr"] Nov 02 14:36:55.521016 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:55.520689 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" podUID="861a94e0-defc-4d04-b11d-ecbdc66002b2" containerName="maas-api" containerID="cri-o://2a8bc16aa21af53b3845151b1aeb18e37950195431a31212ade96927b435a4d0" gracePeriod=30 Nov 02 14:36:55.825876 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:55.825848 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:36:55.994819 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:55.994784 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/861a94e0-defc-4d04-b11d-ecbdc66002b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/861a94e0-defc-4d04-b11d-ecbdc66002b2-maas-api-tls\") pod \"861a94e0-defc-4d04-b11d-ecbdc66002b2\" (UID: \"861a94e0-defc-4d04-b11d-ecbdc66002b2\") " Nov 02 14:36:55.995038 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:55.994864 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/861a94e0-defc-4d04-b11d-ecbdc66002b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/861a94e0-defc-4d04-b11d-ecbdc66002b2-metrics-certs\") pod \"861a94e0-defc-4d04-b11d-ecbdc66002b2\" (UID: \"861a94e0-defc-4d04-b11d-ecbdc66002b2\") " Nov 02 14:36:55.995038 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:55.994959 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/861a94e0-defc-4d04-b11d-ecbdc66002b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/861a94e0-defc-4d04-b11d-ecbdc66002b2-openshift-service-ca\") pod \"861a94e0-defc-4d04-b11d-ecbdc66002b2\" (UID: \"861a94e0-defc-4d04-b11d-ecbdc66002b2\") " Nov 02 14:36:55.995165 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:55.995049 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/861a94e0-defc-4d04-b11d-ecbdc66002b2-kube-api-access-7l9pf\" (UniqueName: \"kubernetes.io/projected/861a94e0-defc-4d04-b11d-ecbdc66002b2-kube-api-access-7l9pf\") pod \"861a94e0-defc-4d04-b11d-ecbdc66002b2\" (UID: \"861a94e0-defc-4d04-b11d-ecbdc66002b2\") " Nov 02 14:36:55.995545 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:55.995383 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/861a94e0-defc-4d04-b11d-ecbdc66002b2-openshift-service-ca" pod "861a94e0-defc-4d04-b11d-ecbdc66002b2" (UID: "861a94e0-defc-4d04-b11d-ecbdc66002b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:55.995545 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:55.995518 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/861a94e0-defc-4d04-b11d-ecbdc66002b2-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:55.997662 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:55.997634 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861a94e0-defc-4d04-b11d-ecbdc66002b2-kube-api-access-7l9pf" pod "861a94e0-defc-4d04-b11d-ecbdc66002b2" (UID: "861a94e0-defc-4d04-b11d-ecbdc66002b2"). InnerVolumeSpecName "kube-api-access-7l9pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:55.997753 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:55.997657 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/861a94e0-defc-4d04-b11d-ecbdc66002b2-metrics-certs" pod "861a94e0-defc-4d04-b11d-ecbdc66002b2" (UID: "861a94e0-defc-4d04-b11d-ecbdc66002b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:36:55.997753 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:55.997699 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/861a94e0-defc-4d04-b11d-ecbdc66002b2-maas-api-tls" pod "861a94e0-defc-4d04-b11d-ecbdc66002b2" (UID: "861a94e0-defc-4d04-b11d-ecbdc66002b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:36:56.099024 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:56.096939 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l9pf\" (UniqueName: \"kubernetes.io/projected/861a94e0-defc-4d04-b11d-ecbdc66002b2-kube-api-access-7l9pf\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:56.099024 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:56.097002 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/861a94e0-defc-4d04-b11d-ecbdc66002b2-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:56.099024 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:56.097018 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/861a94e0-defc-4d04-b11d-ecbdc66002b2-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:36:56.385196 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:56.385112 4866 generic.go:356] "Generic (PLEG): container finished" podID="861a94e0-defc-4d04-b11d-ecbdc66002b2" containerID="2a8bc16aa21af53b3845151b1aeb18e37950195431a31212ade96927b435a4d0" exitCode=0 Nov 02 14:36:56.385606 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:56.385297 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" event={"ID":"861a94e0-defc-4d04-b11d-ecbdc66002b2","Type":"ContainerDied","Data":"2a8bc16aa21af53b3845151b1aeb18e37950195431a31212ade96927b435a4d0"} Nov 02 14:36:56.385782 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:56.385761 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" event={"ID":"861a94e0-defc-4d04-b11d-ecbdc66002b2","Type":"ContainerDied","Data":"542e0f8093cf0d76f73039103aa688086597157c699f7b66caec63be0aa81546"} Nov 02 14:36:56.385920 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:56.385904 4866 scope.go:122] "RemoveContainer" containerID="2a8bc16aa21af53b3845151b1aeb18e37950195431a31212ade96927b435a4d0" Nov 02 14:36:56.386132 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:56.385377 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr" Nov 02 14:36:56.415597 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:56.415571 4866 scope.go:122] "RemoveContainer" containerID="fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991" Nov 02 14:36:56.428685 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:56.428584 4866 scope.go:122] "RemoveContainer" containerID="2a8bc16aa21af53b3845151b1aeb18e37950195431a31212ade96927b435a4d0" Nov 02 14:36:56.429235 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:36:56.429190 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a8bc16aa21af53b3845151b1aeb18e37950195431a31212ade96927b435a4d0\": container with ID starting with 2a8bc16aa21af53b3845151b1aeb18e37950195431a31212ade96927b435a4d0 not found: ID does not exist" containerID="2a8bc16aa21af53b3845151b1aeb18e37950195431a31212ade96927b435a4d0" Nov 02 14:36:56.429346 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:56.429232 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8bc16aa21af53b3845151b1aeb18e37950195431a31212ade96927b435a4d0"} err="failed to get container status \"2a8bc16aa21af53b3845151b1aeb18e37950195431a31212ade96927b435a4d0\": rpc error: code = NotFound desc = could not find container \"2a8bc16aa21af53b3845151b1aeb18e37950195431a31212ade96927b435a4d0\": container with ID starting with 2a8bc16aa21af53b3845151b1aeb18e37950195431a31212ade96927b435a4d0 not found: ID does not exist" Nov 02 14:36:56.429346 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:56.429258 4866 scope.go:122] "RemoveContainer" containerID="fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991" Nov 02 14:36:56.429750 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:36:56.429723 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991\": container with ID starting with fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991 not found: ID does not exist" containerID="fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991" Nov 02 14:36:56.430345 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:56.429922 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991"} err="failed to get container status \"fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991\": rpc error: code = NotFound desc = could not find container \"fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991\": container with ID starting with fce7557e467f21ba81dae919ae5f942212acea9ce7ae63167515025ec064e991 not found: ID does not exist" Nov 02 14:36:56.447582 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:56.447541 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr"] Nov 02 14:36:56.460853 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:56.460814 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4be645-6cd878986-djglr"] Nov 02 14:36:56.954825 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:36:56.954783 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="861a94e0-defc-4d04-b11d-ecbdc66002b2" path="/var/lib/kubelet/pods/861a94e0-defc-4d04-b11d-ecbdc66002b2/volumes" Nov 02 14:37:06.013406 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:37:06.013363 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:37:15.546292 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:15.546258 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:37:15.564015 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:15.563989 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:37:33.969125 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:33.969078 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-8q5hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:37:33.969755 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:33.969719 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" podUID="1fbf701c-a1aa-4031-8112-4e69e00e103b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:37:36.017163 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:37:36.017121 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:37:49.690292 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:49.690257 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf"] Nov 02 14:37:49.690817 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:49.690454 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" podUID="07de4ac7-3fc8-4416-9464-5ec3144a9e03" containerName="discovery" containerID="cri-o://72e8f2fa4674da6782cdc713d37ebf48efacb98391fdc51727b532447dfd0842" gracePeriod=30 Nov 02 14:37:49.837485 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:49.837449 4866 generic.go:356] "Generic (PLEG): container finished" podID="07de4ac7-3fc8-4416-9464-5ec3144a9e03" containerID="72e8f2fa4674da6782cdc713d37ebf48efacb98391fdc51727b532447dfd0842" exitCode=0 Nov 02 14:37:49.837667 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:49.837523 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" event={"ID":"07de4ac7-3fc8-4416-9464-5ec3144a9e03","Type":"ContainerDied","Data":"72e8f2fa4674da6782cdc713d37ebf48efacb98391fdc51727b532447dfd0842"} Nov 02 14:37:49.852610 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:49.852577 4866 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rdlmf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:8080/ready\": dial tcp 10.132.0.75:8080: connect: connection refused" start-of-body= Nov 02 14:37:49.852801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:49.852641 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" podUID="07de4ac7-3fc8-4416-9464-5ec3144a9e03" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.75:8080/ready\": dial tcp 10.132.0.75:8080: connect: connection refused" Nov 02 14:37:49.989975 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:49.989918 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:37:50.132892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.129336 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-csr-ca-configmap\") pod \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " Nov 02 14:37:50.132892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.129461 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07de4ac7-3fc8-4416-9464-5ec3144a9e03-kube-api-access-k62dn\" (UniqueName: \"kubernetes.io/projected/07de4ac7-3fc8-4416-9464-5ec3144a9e03-kube-api-access-k62dn\") pod \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " Nov 02 14:37:50.132892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.129520 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07de4ac7-3fc8-4416-9464-5ec3144a9e03-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07de4ac7-3fc8-4416-9464-5ec3144a9e03-local-certs\") pod \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " Nov 02 14:37:50.132892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.129555 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-token\" (UniqueName: \"kubernetes.io/projected/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-token\") pod \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " Nov 02 14:37:50.132892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.129656 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-cacerts\" (UniqueName: \"kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-cacerts\") pod \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " Nov 02 14:37:50.132892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.129735 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-csr-dns-cert\") pod \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " Nov 02 14:37:50.132892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.129830 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-kubeconfig\") pod \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\" (UID: \"07de4ac7-3fc8-4416-9464-5ec3144a9e03\") " Nov 02 14:37:50.134395 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.133756 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-kubeconfig" pod "07de4ac7-3fc8-4416-9464-5ec3144a9e03" (UID: "07de4ac7-3fc8-4416-9464-5ec3144a9e03"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:37:50.134395 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.134103 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-csr-ca-configmap" pod "07de4ac7-3fc8-4416-9464-5ec3144a9e03" (UID: "07de4ac7-3fc8-4416-9464-5ec3144a9e03"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:37:50.136900 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.136637 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07de4ac7-3fc8-4416-9464-5ec3144a9e03-kube-api-access-k62dn" pod "07de4ac7-3fc8-4416-9464-5ec3144a9e03" (UID: "07de4ac7-3fc8-4416-9464-5ec3144a9e03"). InnerVolumeSpecName "kube-api-access-k62dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:37:50.138929 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.138902 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07de4ac7-3fc8-4416-9464-5ec3144a9e03-local-certs" pod "07de4ac7-3fc8-4416-9464-5ec3144a9e03" (UID: "07de4ac7-3fc8-4416-9464-5ec3144a9e03"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:37:50.140406 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.140380 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-cacerts" pod "07de4ac7-3fc8-4416-9464-5ec3144a9e03" (UID: "07de4ac7-3fc8-4416-9464-5ec3144a9e03"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:37:50.140842 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.140820 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-token" pod "07de4ac7-3fc8-4416-9464-5ec3144a9e03" (UID: "07de4ac7-3fc8-4416-9464-5ec3144a9e03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:37:50.140911 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.140839 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-csr-dns-cert" pod "07de4ac7-3fc8-4416-9464-5ec3144a9e03" (UID: "07de4ac7-3fc8-4416-9464-5ec3144a9e03"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:37:50.231561 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.231448 4866 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-kubeconfig\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:37:50.231561 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.231497 4866 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-csr-ca-configmap\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:37:50.231561 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.231512 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k62dn\" (UniqueName: \"kubernetes.io/projected/07de4ac7-3fc8-4416-9464-5ec3144a9e03-kube-api-access-k62dn\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:37:50.231561 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.231524 4866 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07de4ac7-3fc8-4416-9464-5ec3144a9e03-local-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:37:50.231561 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.231536 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:37:50.231561 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.231549 4866 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-cacerts\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:37:50.231561 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.231560 4866 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07de4ac7-3fc8-4416-9464-5ec3144a9e03-istio-csr-dns-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:37:50.845346 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.845314 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" Nov 02 14:37:50.845346 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.845329 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf" event={"ID":"07de4ac7-3fc8-4416-9464-5ec3144a9e03","Type":"ContainerDied","Data":"459c3af81bac6aeb006b448eaf2f2c89d750dd98b20d1ee7f501c1aa42034f89"} Nov 02 14:37:50.845903 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.845427 4866 scope.go:122] "RemoveContainer" containerID="72e8f2fa4674da6782cdc713d37ebf48efacb98391fdc51727b532447dfd0842" Nov 02 14:37:50.911930 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.911897 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf"] Nov 02 14:37:50.923068 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.923038 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdlmf"] Nov 02 14:37:50.946849 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:50.946814 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07de4ac7-3fc8-4416-9464-5ec3144a9e03" path="/var/lib/kubelet/pods/07de4ac7-3fc8-4416-9464-5ec3144a9e03/volumes" Nov 02 14:37:57.512589 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:57.512549 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m"] Nov 02 14:37:57.513131 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:37:57.512775 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" podUID="a9ae3055-5b22-4b7e-bee4-d853a0b8517b" containerName="istio-proxy" containerID="cri-o://092da1b100bd2d58520e27e9a026b8078fdb875e0fa0badbe16e61371a60b1f8" gracePeriod=30 Nov 02 14:38:00.405306 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:00.405267 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc"] Nov 02 14:38:00.405767 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:00.405526 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" podUID="de984a72-1f11-42aa-a50a-1013f70af8f9" containerName="istio-proxy" containerID="cri-o://5a70d63d7b8eb7d874977324f633256203421e6014092560612136da9d298fb8" gracePeriod=30 Nov 02 14:38:02.784346 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.784272 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:38:02.792265 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.792239 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-workload-socket\") pod \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " Nov 02 14:38:02.792400 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.792313 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istiod-ca-cert\") pod \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " Nov 02 14:38:02.792400 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.792360 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-credential-socket\") pod \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " Nov 02 14:38:02.792400 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.792394 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-token\" (UniqueName: \"kubernetes.io/projected/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-token\") pod \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " Nov 02 14:38:02.792569 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.792503 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-podinfo\") pod \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " Nov 02 14:38:02.792569 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.792536 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-envoy\") pod \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " Nov 02 14:38:02.792673 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.792574 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-data\") pod \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " Nov 02 14:38:02.792673 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.792558 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-workload-socket" pod "a9ae3055-5b22-4b7e-bee4-d853a0b8517b" (UID: "a9ae3055-5b22-4b7e-bee4-d853a0b8517b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:38:02.792673 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.792657 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-credential-socket" pod "a9ae3055-5b22-4b7e-bee4-d853a0b8517b" (UID: "a9ae3055-5b22-4b7e-bee4-d853a0b8517b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:38:02.792845 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.792666 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-kube-api-access-g82fz\" (UniqueName: \"kubernetes.io/projected/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-kube-api-access-g82fz\") pod \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " Nov 02 14:38:02.792845 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.792724 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-workload-certs\") pod \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\" (UID: \"a9ae3055-5b22-4b7e-bee4-d853a0b8517b\") " Nov 02 14:38:02.792845 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.792828 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-data" pod "a9ae3055-5b22-4b7e-bee4-d853a0b8517b" (UID: "a9ae3055-5b22-4b7e-bee4-d853a0b8517b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:38:02.793035 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.792855 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istiod-ca-cert" pod "a9ae3055-5b22-4b7e-bee4-d853a0b8517b" (UID: "a9ae3055-5b22-4b7e-bee4-d853a0b8517b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:38:02.793108 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.793087 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-workload-certs" pod "a9ae3055-5b22-4b7e-bee4-d853a0b8517b" (UID: "a9ae3055-5b22-4b7e-bee4-d853a0b8517b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:38:02.793276 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.793251 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:02.793400 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.793281 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:02.793400 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.793295 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:02.793400 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.793308 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:02.793400 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.793321 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:02.794801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.794775 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-envoy" pod "a9ae3055-5b22-4b7e-bee4-d853a0b8517b" (UID: "a9ae3055-5b22-4b7e-bee4-d853a0b8517b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:38:02.794890 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.794806 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-kube-api-access-g82fz" pod "a9ae3055-5b22-4b7e-bee4-d853a0b8517b" (UID: "a9ae3055-5b22-4b7e-bee4-d853a0b8517b"). InnerVolumeSpecName "kube-api-access-g82fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:38:02.795117 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.795096 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-podinfo" pod "a9ae3055-5b22-4b7e-bee4-d853a0b8517b" (UID: "a9ae3055-5b22-4b7e-bee4-d853a0b8517b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:38:02.795183 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.795122 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-token" pod "a9ae3055-5b22-4b7e-bee4-d853a0b8517b" (UID: "a9ae3055-5b22-4b7e-bee4-d853a0b8517b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:38:02.894618 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.894574 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:02.894618 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.894618 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:02.894841 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.894632 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g82fz\" (UniqueName: \"kubernetes.io/projected/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-kube-api-access-g82fz\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:02.894841 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.894646 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ae3055-5b22-4b7e-bee4-d853a0b8517b-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:02.922804 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.922768 4866 generic.go:356] "Generic (PLEG): container finished" podID="a9ae3055-5b22-4b7e-bee4-d853a0b8517b" containerID="092da1b100bd2d58520e27e9a026b8078fdb875e0fa0badbe16e61371a60b1f8" exitCode=0 Nov 02 14:38:02.923024 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.922823 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" Nov 02 14:38:02.923024 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.922859 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" event={"ID":"a9ae3055-5b22-4b7e-bee4-d853a0b8517b","Type":"ContainerDied","Data":"092da1b100bd2d58520e27e9a026b8078fdb875e0fa0badbe16e61371a60b1f8"} Nov 02 14:38:02.923024 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.922920 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m" event={"ID":"a9ae3055-5b22-4b7e-bee4-d853a0b8517b","Type":"ContainerDied","Data":"3076c42167014cd71d21c6bc679f786ed6ab7ce4601f7686a05a1348f2eab0f8"} Nov 02 14:38:02.923024 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.922969 4866 scope.go:122] "RemoveContainer" containerID="092da1b100bd2d58520e27e9a026b8078fdb875e0fa0badbe16e61371a60b1f8" Nov 02 14:38:02.933492 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.933469 4866 scope.go:122] "RemoveContainer" containerID="092da1b100bd2d58520e27e9a026b8078fdb875e0fa0badbe16e61371a60b1f8" Nov 02 14:38:02.933800 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:38:02.933780 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"092da1b100bd2d58520e27e9a026b8078fdb875e0fa0badbe16e61371a60b1f8\": container with ID starting with 092da1b100bd2d58520e27e9a026b8078fdb875e0fa0badbe16e61371a60b1f8 not found: ID does not exist" containerID="092da1b100bd2d58520e27e9a026b8078fdb875e0fa0badbe16e61371a60b1f8" Nov 02 14:38:02.933859 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.933810 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092da1b100bd2d58520e27e9a026b8078fdb875e0fa0badbe16e61371a60b1f8"} err="failed to get container status \"092da1b100bd2d58520e27e9a026b8078fdb875e0fa0badbe16e61371a60b1f8\": rpc error: code = NotFound desc = could not find container \"092da1b100bd2d58520e27e9a026b8078fdb875e0fa0badbe16e61371a60b1f8\": container with ID starting with 092da1b100bd2d58520e27e9a026b8078fdb875e0fa0badbe16e61371a60b1f8 not found: ID does not exist" Nov 02 14:38:02.987012 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.986974 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m"] Nov 02 14:38:02.991281 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:02.991250 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-sbv6m"] Nov 02 14:38:04.945195 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:04.945163 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9ae3055-5b22-4b7e-bee4-d853a0b8517b" path="/var/lib/kubelet/pods/a9ae3055-5b22-4b7e-bee4-d853a0b8517b/volumes" Nov 02 14:38:05.514799 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.514764 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 14:38:05.515006 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.514829 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" podUID="de984a72-1f11-42aa-a50a-1013f70af8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 14:38:05.678831 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.678768 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:38:05.725641 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.725609 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de984a72-1f11-42aa-a50a-1013f70af8f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de984a72-1f11-42aa-a50a-1013f70af8f9-istiod-ca-cert\") pod \"de984a72-1f11-42aa-a50a-1013f70af8f9\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " Nov 02 14:38:05.725823 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.725684 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-workload-certs\") pod \"de984a72-1f11-42aa-a50a-1013f70af8f9\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " Nov 02 14:38:05.725823 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.725736 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-envoy\") pod \"de984a72-1f11-42aa-a50a-1013f70af8f9\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " Nov 02 14:38:05.725823 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.725754 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-podinfo\") pod \"de984a72-1f11-42aa-a50a-1013f70af8f9\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " Nov 02 14:38:05.725823 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.725780 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de984a72-1f11-42aa-a50a-1013f70af8f9-kube-api-access-mqckh\" (UniqueName: \"kubernetes.io/projected/de984a72-1f11-42aa-a50a-1013f70af8f9-kube-api-access-mqckh\") pod \"de984a72-1f11-42aa-a50a-1013f70af8f9\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " Nov 02 14:38:05.725823 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.725805 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-credential-socket\") pod \"de984a72-1f11-42aa-a50a-1013f70af8f9\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " Nov 02 14:38:05.726113 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.725928 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-token\" (UniqueName: \"kubernetes.io/projected/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-token\") pod \"de984a72-1f11-42aa-a50a-1013f70af8f9\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " Nov 02 14:38:05.726113 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.726013 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-workload-socket\") pod \"de984a72-1f11-42aa-a50a-1013f70af8f9\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " Nov 02 14:38:05.726113 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.726054 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-data\") pod \"de984a72-1f11-42aa-a50a-1013f70af8f9\" (UID: \"de984a72-1f11-42aa-a50a-1013f70af8f9\") " Nov 02 14:38:05.726113 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.726079 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de984a72-1f11-42aa-a50a-1013f70af8f9-istiod-ca-cert" pod "de984a72-1f11-42aa-a50a-1013f70af8f9" (UID: "de984a72-1f11-42aa-a50a-1013f70af8f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:38:05.726113 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.726092 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-workload-certs" pod "de984a72-1f11-42aa-a50a-1013f70af8f9" (UID: "de984a72-1f11-42aa-a50a-1013f70af8f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:38:05.726453 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.726309 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-credential-socket" pod "de984a72-1f11-42aa-a50a-1013f70af8f9" (UID: "de984a72-1f11-42aa-a50a-1013f70af8f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:38:05.726453 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.726346 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-workload-socket" pod "de984a72-1f11-42aa-a50a-1013f70af8f9" (UID: "de984a72-1f11-42aa-a50a-1013f70af8f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:38:05.726557 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.726513 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-data" pod "de984a72-1f11-42aa-a50a-1013f70af8f9" (UID: "de984a72-1f11-42aa-a50a-1013f70af8f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:38:05.726640 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.726625 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:05.726682 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.726647 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:05.726682 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.726660 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de984a72-1f11-42aa-a50a-1013f70af8f9-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:05.726682 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.726672 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:05.726775 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.726685 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:05.728225 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.728197 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de984a72-1f11-42aa-a50a-1013f70af8f9-kube-api-access-mqckh" pod "de984a72-1f11-42aa-a50a-1013f70af8f9" (UID: "de984a72-1f11-42aa-a50a-1013f70af8f9"). InnerVolumeSpecName "kube-api-access-mqckh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:38:05.728225 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.728212 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-token" pod "de984a72-1f11-42aa-a50a-1013f70af8f9" (UID: "de984a72-1f11-42aa-a50a-1013f70af8f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:38:05.728436 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.728419 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-podinfo" pod "de984a72-1f11-42aa-a50a-1013f70af8f9" (UID: "de984a72-1f11-42aa-a50a-1013f70af8f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:38:05.728496 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.728431 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-envoy" pod "de984a72-1f11-42aa-a50a-1013f70af8f9" (UID: "de984a72-1f11-42aa-a50a-1013f70af8f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:38:05.827554 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.827511 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:05.827554 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.827544 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:05.827554 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.827555 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqckh\" (UniqueName: \"kubernetes.io/projected/de984a72-1f11-42aa-a50a-1013f70af8f9-kube-api-access-mqckh\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:05.827554 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.827565 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de984a72-1f11-42aa-a50a-1013f70af8f9-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:05.942293 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.942260 4866 generic.go:356] "Generic (PLEG): container finished" podID="de984a72-1f11-42aa-a50a-1013f70af8f9" containerID="5a70d63d7b8eb7d874977324f633256203421e6014092560612136da9d298fb8" exitCode=0 Nov 02 14:38:05.942495 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.942317 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" Nov 02 14:38:05.942495 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.942320 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" event={"ID":"de984a72-1f11-42aa-a50a-1013f70af8f9","Type":"ContainerDied","Data":"5a70d63d7b8eb7d874977324f633256203421e6014092560612136da9d298fb8"} Nov 02 14:38:05.942495 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.942436 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc" event={"ID":"de984a72-1f11-42aa-a50a-1013f70af8f9","Type":"ContainerDied","Data":"60851121c6617d0c6922c43bd119710b18c83ce147e6acc04bde7ed462919c52"} Nov 02 14:38:05.942495 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.942460 4866 scope.go:122] "RemoveContainer" containerID="5a70d63d7b8eb7d874977324f633256203421e6014092560612136da9d298fb8" Nov 02 14:38:05.952825 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.952800 4866 scope.go:122] "RemoveContainer" containerID="5a70d63d7b8eb7d874977324f633256203421e6014092560612136da9d298fb8" Nov 02 14:38:05.953323 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:38:05.953166 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a70d63d7b8eb7d874977324f633256203421e6014092560612136da9d298fb8\": container with ID starting with 5a70d63d7b8eb7d874977324f633256203421e6014092560612136da9d298fb8 not found: ID does not exist" containerID="5a70d63d7b8eb7d874977324f633256203421e6014092560612136da9d298fb8" Nov 02 14:38:05.953323 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.953217 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a70d63d7b8eb7d874977324f633256203421e6014092560612136da9d298fb8"} err="failed to get container status \"5a70d63d7b8eb7d874977324f633256203421e6014092560612136da9d298fb8\": rpc error: code = NotFound desc = could not find container \"5a70d63d7b8eb7d874977324f633256203421e6014092560612136da9d298fb8\": container with ID starting with 5a70d63d7b8eb7d874977324f633256203421e6014092560612136da9d298fb8 not found: ID does not exist" Nov 02 14:38:05.997806 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:05.997766 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc"] Nov 02 14:38:06.002338 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:06.002310 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9hrdc"] Nov 02 14:38:06.013310 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:38:06.013280 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:38:06.947588 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:06.947551 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de984a72-1f11-42aa-a50a-1013f70af8f9" path="/var/lib/kubelet/pods/de984a72-1f11-42aa-a50a-1013f70af8f9/volumes" Nov 02 14:38:26.365357 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:26.365320 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6"] Nov 02 14:38:36.010907 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:38:36.010815 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:38:39.853670 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.853633 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7813-09c805"] Nov 02 14:38:39.854264 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854246 4866 cpu_manager.go:414] "containerMap: removing container" podUID="39897b77-0b2d-42b1-9c4f-77cfe4025791" containerName="payload-pre-processing" Nov 02 14:38:39.854309 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854265 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39897b77-0b2d-42b1-9c4f-77cfe4025791" containerName="payload-pre-processing" Nov 02 14:38:39.854309 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854280 4866 cpu_manager.go:414] "containerMap: removing container" podUID="ae683718-049c-4ea8-82da-4f97e31175ac" containerName="payload-pre-processing" Nov 02 14:38:39.854309 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854285 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae683718-049c-4ea8-82da-4f97e31175ac" containerName="payload-pre-processing" Nov 02 14:38:39.854309 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854301 4866 cpu_manager.go:414] "containerMap: removing container" podUID="f9d55902-d242-4c33-98dd-0c273256c497" containerName="payload-processing" Nov 02 14:38:39.854309 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854306 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d55902-d242-4c33-98dd-0c273256c497" containerName="payload-processing" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854315 4866 cpu_manager.go:414] "containerMap: removing container" podUID="861a94e0-defc-4d04-b11d-ecbdc66002b2" containerName="maas-api" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854319 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="861a94e0-defc-4d04-b11d-ecbdc66002b2" containerName="maas-api" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854325 4866 cpu_manager.go:414] "containerMap: removing container" podUID="f06bd94b-7a9c-4463-8d81-1650b45cafdc" containerName="payload-processing" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854329 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06bd94b-7a9c-4463-8d81-1650b45cafdc" containerName="payload-processing" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854338 4866 cpu_manager.go:414] "containerMap: removing container" podUID="4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e" containerName="main" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854342 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e" containerName="main" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854363 4866 cpu_manager.go:414] "containerMap: removing container" podUID="0129df2e-9d21-45f7-9aa2-53777ce753b4" containerName="main" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854367 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0129df2e-9d21-45f7-9aa2-53777ce753b4" containerName="main" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854373 4866 cpu_manager.go:414] "containerMap: removing container" podUID="861a94e0-defc-4d04-b11d-ecbdc66002b2" containerName="maas-api" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854377 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="861a94e0-defc-4d04-b11d-ecbdc66002b2" containerName="maas-api" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854384 4866 cpu_manager.go:414] "containerMap: removing container" podUID="de984a72-1f11-42aa-a50a-1013f70af8f9" containerName="istio-proxy" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854389 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de984a72-1f11-42aa-a50a-1013f70af8f9" containerName="istio-proxy" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854397 4866 cpu_manager.go:414] "containerMap: removing container" podUID="07de4ac7-3fc8-4416-9464-5ec3144a9e03" containerName="discovery" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854402 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07de4ac7-3fc8-4416-9464-5ec3144a9e03" containerName="discovery" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854410 4866 cpu_manager.go:414] "containerMap: removing container" podUID="d73bd157-9369-406e-b2bf-d4cfa38a1d33" containerName="payload-processing" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854414 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73bd157-9369-406e-b2bf-d4cfa38a1d33" containerName="payload-processing" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854425 4866 cpu_manager.go:414] "containerMap: removing container" podUID="a9ae3055-5b22-4b7e-bee4-d853a0b8517b" containerName="istio-proxy" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854429 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ae3055-5b22-4b7e-bee4-d853a0b8517b" containerName="istio-proxy" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854441 4866 cpu_manager.go:414] "containerMap: removing container" podUID="4991d22f-27fd-44d9-bcfe-2f991088c0bc" containerName="payload-processing" Nov 02 14:38:39.854462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854446 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4991d22f-27fd-44d9-bcfe-2f991088c0bc" containerName="payload-processing" Nov 02 14:38:39.855079 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854529 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f9dd65a-ec89-40fb-8fd5-4bae69a0ee7e" containerName="main" Nov 02 14:38:39.855079 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854540 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="0129df2e-9d21-45f7-9aa2-53777ce753b4" containerName="main" Nov 02 14:38:39.855079 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854546 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="861a94e0-defc-4d04-b11d-ecbdc66002b2" containerName="maas-api" Nov 02 14:38:39.855079 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854553 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="de984a72-1f11-42aa-a50a-1013f70af8f9" containerName="istio-proxy" Nov 02 14:38:39.855079 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854559 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="07de4ac7-3fc8-4416-9464-5ec3144a9e03" containerName="discovery" Nov 02 14:38:39.855079 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854565 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73bd157-9369-406e-b2bf-d4cfa38a1d33" containerName="payload-processing" Nov 02 14:38:39.855079 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854570 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ae3055-5b22-4b7e-bee4-d853a0b8517b" containerName="istio-proxy" Nov 02 14:38:39.855079 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854575 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="4991d22f-27fd-44d9-bcfe-2f991088c0bc" containerName="payload-processing" Nov 02 14:38:39.855079 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854581 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="39897b77-0b2d-42b1-9c4f-77cfe4025791" containerName="payload-pre-processing" Nov 02 14:38:39.855079 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854589 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae683718-049c-4ea8-82da-4f97e31175ac" containerName="payload-pre-processing" Nov 02 14:38:39.855079 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854595 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d55902-d242-4c33-98dd-0c273256c497" containerName="payload-processing" Nov 02 14:38:39.855079 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.854600 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06bd94b-7a9c-4463-8d81-1650b45cafdc" containerName="payload-processing" Nov 02 14:38:39.857403 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.857388 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7813-09c805" Nov 02 14:38:39.888279 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.888114 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7813-09c805"] Nov 02 14:38:39.893667 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.893639 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nz2z6\"" Nov 02 14:38:39.956204 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:39.956168 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m6kf\" (UniqueName: \"kubernetes.io/projected/e58e566f-2681-4b33-879e-8ed9331da382-kube-api-access-2m6kf\") pod \"test-curl-7813-09c805\" (UID: \"e58e566f-2681-4b33-879e-8ed9331da382\") " pod="opendatahub/test-curl-7813-09c805" Nov 02 14:38:40.057823 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:40.057781 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m6kf\" (UniqueName: \"kubernetes.io/projected/e58e566f-2681-4b33-879e-8ed9331da382-kube-api-access-2m6kf\") pod \"test-curl-7813-09c805\" (UID: \"e58e566f-2681-4b33-879e-8ed9331da382\") " pod="opendatahub/test-curl-7813-09c805" Nov 02 14:38:40.082280 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:40.082245 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m6kf\" (UniqueName: \"kubernetes.io/projected/e58e566f-2681-4b33-879e-8ed9331da382-kube-api-access-2m6kf\") pod \"test-curl-7813-09c805\" (UID: \"e58e566f-2681-4b33-879e-8ed9331da382\") " pod="opendatahub/test-curl-7813-09c805" Nov 02 14:38:40.170578 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:40.170489 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7813-09c805" Nov 02 14:38:40.484260 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:40.484142 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7813-09c805"] Nov 02 14:38:40.487020 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:38:40.486993 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode58e566f_2681_4b33_879e_8ed9331da382.slice/crio-306e3d622f64066a5763800d4092429534c71b63512fe34a870be2b3c5f51a85 WatchSource:0}: Error finding container 306e3d622f64066a5763800d4092429534c71b63512fe34a870be2b3c5f51a85: Status 404 returned error can't find the container with id 306e3d622f64066a5763800d4092429534c71b63512fe34a870be2b3c5f51a85 Nov 02 14:38:41.183854 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.183814 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7813-09c805" event={"ID":"e58e566f-2681-4b33-879e-8ed9331da382","Type":"ContainerStarted","Data":"6c6901b6363666f790a0e8abf204b6ac0aeba6a1d7937b341eaec72d342ee270"} Nov 02 14:38:41.184394 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.183860 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7813-09c805" event={"ID":"e58e566f-2681-4b33-879e-8ed9331da382","Type":"ContainerStarted","Data":"306e3d622f64066a5763800d4092429534c71b63512fe34a870be2b3c5f51a85"} Nov 02 14:38:41.227157 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.227103 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7813-09c805" podStartSLOduration=2.227089203 podStartE2EDuration="2.227089203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:38:41.22670527 +0000 UTC m=+2486.410552606" watchObservedRunningTime="2026-11-02 14:38:41.227089203 +0000 UTC m=+2486.410936539" Nov 02 14:38:41.386845 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.386802 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" podUID="91e00078-7b8d-41d9-b385-4ee8a4fe3b50" containerName="main" containerID="cri-o://3c224ede1af5ed3379e935ae833083eecdc91d753e72cefa50fe75455a134ff2" gracePeriod=45 Nov 02 14:38:41.659325 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.659297 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:38:41.775193 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.775094 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-home\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-home\") pod \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " Nov 02 14:38:41.775193 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.775173 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-model-cache\") pod \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " Nov 02 14:38:41.775440 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.775204 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-kube-api-access-4xndg\" (UniqueName: \"kubernetes.io/projected/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-kube-api-access-4xndg\") pod \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " Nov 02 14:38:41.775440 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.775317 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-dshm\") pod \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " Nov 02 14:38:41.775440 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.775376 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-tls-certs\" (UniqueName: \"kubernetes.io/secret/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-tls-certs\") pod \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " Nov 02 14:38:41.775440 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.775427 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-home" pod "91e00078-7b8d-41d9-b385-4ee8a4fe3b50" (UID: "91e00078-7b8d-41d9-b385-4ee8a4fe3b50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:38:41.775626 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.775451 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-tmp-dir\") pod \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\" (UID: \"91e00078-7b8d-41d9-b385-4ee8a4fe3b50\") " Nov 02 14:38:41.776011 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.775918 4866 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-home\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:41.776213 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.776190 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-model-cache" pod "91e00078-7b8d-41d9-b385-4ee8a4fe3b50" (UID: "91e00078-7b8d-41d9-b385-4ee8a4fe3b50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:38:41.776432 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.776411 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-tmp-dir" pod "91e00078-7b8d-41d9-b385-4ee8a4fe3b50" (UID: "91e00078-7b8d-41d9-b385-4ee8a4fe3b50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:38:41.778530 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.778500 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-dshm" pod "91e00078-7b8d-41d9-b385-4ee8a4fe3b50" (UID: "91e00078-7b8d-41d9-b385-4ee8a4fe3b50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:38:41.778640 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.778582 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-kube-api-access-4xndg" pod "91e00078-7b8d-41d9-b385-4ee8a4fe3b50" (UID: "91e00078-7b8d-41d9-b385-4ee8a4fe3b50"). InnerVolumeSpecName "kube-api-access-4xndg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:38:41.778640 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.778583 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-tls-certs" pod "91e00078-7b8d-41d9-b385-4ee8a4fe3b50" (UID: "91e00078-7b8d-41d9-b385-4ee8a4fe3b50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:38:41.877274 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.877236 4866 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-tmp-dir\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:41.877450 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.877279 4866 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-model-cache\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:41.877450 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.877296 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xndg\" (UniqueName: \"kubernetes.io/projected/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-kube-api-access-4xndg\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:41.877450 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.877310 4866 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-dshm\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:41.877450 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:41.877325 4866 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e00078-7b8d-41d9-b385-4ee8a4fe3b50-tls-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:42.168803 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.168719 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7813-09c805"] Nov 02 14:38:42.180202 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.180171 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7813-09c805"] Nov 02 14:38:42.192221 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.192185 4866 generic.go:356] "Generic (PLEG): container finished" podID="91e00078-7b8d-41d9-b385-4ee8a4fe3b50" containerID="3c224ede1af5ed3379e935ae833083eecdc91d753e72cefa50fe75455a134ff2" exitCode=0 Nov 02 14:38:42.192733 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.192237 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" Nov 02 14:38:42.192733 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.192286 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" event={"ID":"91e00078-7b8d-41d9-b385-4ee8a4fe3b50","Type":"ContainerDied","Data":"3c224ede1af5ed3379e935ae833083eecdc91d753e72cefa50fe75455a134ff2"} Nov 02 14:38:42.192733 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.192340 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6" event={"ID":"91e00078-7b8d-41d9-b385-4ee8a4fe3b50","Type":"ContainerDied","Data":"69a919f8fcfb523ad4b8d619bdcbe3e1cf8c7ed3466324da07f871cc1e194cce"} Nov 02 14:38:42.192733 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.192364 4866 scope.go:122] "RemoveContainer" containerID="3c224ede1af5ed3379e935ae833083eecdc91d753e72cefa50fe75455a134ff2" Nov 02 14:38:42.205193 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.205171 4866 scope.go:122] "RemoveContainer" containerID="3c224ede1af5ed3379e935ae833083eecdc91d753e72cefa50fe75455a134ff2" Nov 02 14:38:42.205530 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:38:42.205505 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c224ede1af5ed3379e935ae833083eecdc91d753e72cefa50fe75455a134ff2\": container with ID starting with 3c224ede1af5ed3379e935ae833083eecdc91d753e72cefa50fe75455a134ff2 not found: ID does not exist" containerID="3c224ede1af5ed3379e935ae833083eecdc91d753e72cefa50fe75455a134ff2" Nov 02 14:38:42.205597 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.205542 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c224ede1af5ed3379e935ae833083eecdc91d753e72cefa50fe75455a134ff2"} err="failed to get container status \"3c224ede1af5ed3379e935ae833083eecdc91d753e72cefa50fe75455a134ff2\": rpc error: code = NotFound desc = could not find container \"3c224ede1af5ed3379e935ae833083eecdc91d753e72cefa50fe75455a134ff2\": container with ID starting with 3c224ede1af5ed3379e935ae833083eecdc91d753e72cefa50fe75455a134ff2 not found: ID does not exist" Nov 02 14:38:42.268459 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.268425 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6"] Nov 02 14:38:42.273826 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.273795 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-623b4b/rate-test-model-623b4b-kserve-c444f69d5-qmrh6"] Nov 02 14:38:42.352641 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.352604 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7813-b1ec9c"] Nov 02 14:38:42.353170 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.353156 4866 cpu_manager.go:414] "containerMap: removing container" podUID="e58e566f-2681-4b33-879e-8ed9331da382" containerName="test-curl-7813-09c805" Nov 02 14:38:42.353219 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.353172 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58e566f-2681-4b33-879e-8ed9331da382" containerName="test-curl-7813-09c805" Nov 02 14:38:42.353219 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.353186 4866 cpu_manager.go:414] "containerMap: removing container" podUID="91e00078-7b8d-41d9-b385-4ee8a4fe3b50" containerName="main" Nov 02 14:38:42.353219 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.353191 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e00078-7b8d-41d9-b385-4ee8a4fe3b50" containerName="main" Nov 02 14:38:42.353313 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.353298 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="861a94e0-defc-4d04-b11d-ecbdc66002b2" containerName="maas-api" Nov 02 14:38:42.353313 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.353308 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58e566f-2681-4b33-879e-8ed9331da382" containerName="test-curl-7813-09c805" Nov 02 14:38:42.353374 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.353315 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e00078-7b8d-41d9-b385-4ee8a4fe3b50" containerName="main" Nov 02 14:38:42.357824 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.357806 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7813-b1ec9c" Nov 02 14:38:42.381841 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.381805 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zql2k\" (UniqueName: \"kubernetes.io/projected/39907e49-1f0d-4a5b-a4b3-f44d93564ed2-kube-api-access-zql2k\") pod \"test-curl-7813-b1ec9c\" (UID: \"39907e49-1f0d-4a5b-a4b3-f44d93564ed2\") " pod="opendatahub/test-curl-7813-b1ec9c" Nov 02 14:38:42.386816 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.386770 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7813-b1ec9c"] Nov 02 14:38:42.482690 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.482654 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zql2k\" (UniqueName: \"kubernetes.io/projected/39907e49-1f0d-4a5b-a4b3-f44d93564ed2-kube-api-access-zql2k\") pod \"test-curl-7813-b1ec9c\" (UID: \"39907e49-1f0d-4a5b-a4b3-f44d93564ed2\") " pod="opendatahub/test-curl-7813-b1ec9c" Nov 02 14:38:42.537135 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.537107 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zql2k\" (UniqueName: \"kubernetes.io/projected/39907e49-1f0d-4a5b-a4b3-f44d93564ed2-kube-api-access-zql2k\") pod \"test-curl-7813-b1ec9c\" (UID: \"39907e49-1f0d-4a5b-a4b3-f44d93564ed2\") " pod="opendatahub/test-curl-7813-b1ec9c" Nov 02 14:38:42.674711 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.674677 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7813-b1ec9c" Nov 02 14:38:42.945910 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.945883 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e00078-7b8d-41d9-b385-4ee8a4fe3b50" path="/var/lib/kubelet/pods/91e00078-7b8d-41d9-b385-4ee8a4fe3b50/volumes" Nov 02 14:38:42.974213 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:42.974188 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7813-b1ec9c"] Nov 02 14:38:42.975824 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:38:42.975793 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39907e49_1f0d_4a5b_a4b3_f44d93564ed2.slice/crio-1da725eee120f2c622db22eb15188a4204ef314b499610eb342a99a6d4825417 WatchSource:0}: Error finding container 1da725eee120f2c622db22eb15188a4204ef314b499610eb342a99a6d4825417: Status 404 returned error can't find the container with id 1da725eee120f2c622db22eb15188a4204ef314b499610eb342a99a6d4825417 Nov 02 14:38:43.202409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:43.202315 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7813-b1ec9c" event={"ID":"39907e49-1f0d-4a5b-a4b3-f44d93564ed2","Type":"ContainerStarted","Data":"22c1ceadb678f74180f42907b5cd796f5422184de6cb480e80b1b120f7eb07c8"} Nov 02 14:38:43.202409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:43.202361 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7813-b1ec9c" event={"ID":"39907e49-1f0d-4a5b-a4b3-f44d93564ed2","Type":"ContainerStarted","Data":"1da725eee120f2c622db22eb15188a4204ef314b499610eb342a99a6d4825417"} Nov 02 14:38:43.202409 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:43.202391 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7813-09c805" podUID="e58e566f-2681-4b33-879e-8ed9331da382" containerName="test-curl-7813-09c805" containerID="cri-o://6c6901b6363666f790a0e8abf204b6ac0aeba6a1d7937b341eaec72d342ee270" gracePeriod=2 Nov 02 14:38:43.247902 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:43.247738 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7813-b1ec9c" podStartSLOduration=1.247719614 podStartE2EDuration="1.247719614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:38:43.246165725 +0000 UTC m=+2488.430013062" watchObservedRunningTime="2026-11-02 14:38:43.247719614 +0000 UTC m=+2488.431566954" Nov 02 14:38:44.127194 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:44.127155 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7813-b1ec9c"] Nov 02 14:38:44.133090 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:44.133055 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7813-b1ec9c"] Nov 02 14:38:45.213199 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:45.213163 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7813-b1ec9c" podUID="39907e49-1f0d-4a5b-a4b3-f44d93564ed2" containerName="test-curl-7813-b1ec9c" containerID="cri-o://22c1ceadb678f74180f42907b5cd796f5422184de6cb480e80b1b120f7eb07c8" gracePeriod=2 Nov 02 14:38:45.402438 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:45.402408 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7813-09c805" Nov 02 14:38:45.414281 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:45.414250 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58e566f-2681-4b33-879e-8ed9331da382-kube-api-access-2m6kf\" (UniqueName: \"kubernetes.io/projected/e58e566f-2681-4b33-879e-8ed9331da382-kube-api-access-2m6kf\") pod \"e58e566f-2681-4b33-879e-8ed9331da382\" (UID: \"e58e566f-2681-4b33-879e-8ed9331da382\") " Nov 02 14:38:45.416373 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:45.416346 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58e566f-2681-4b33-879e-8ed9331da382-kube-api-access-2m6kf" pod "e58e566f-2681-4b33-879e-8ed9331da382" (UID: "e58e566f-2681-4b33-879e-8ed9331da382"). InnerVolumeSpecName "kube-api-access-2m6kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:38:45.515815 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:45.515731 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m6kf\" (UniqueName: \"kubernetes.io/projected/e58e566f-2681-4b33-879e-8ed9331da382-kube-api-access-2m6kf\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:46.220166 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:46.220134 4866 generic.go:356] "Generic (PLEG): container finished" podID="e58e566f-2681-4b33-879e-8ed9331da382" containerID="6c6901b6363666f790a0e8abf204b6ac0aeba6a1d7937b341eaec72d342ee270" exitCode=137 Nov 02 14:38:46.220607 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:46.220206 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7813-09c805" Nov 02 14:38:46.220607 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:46.220256 4866 scope.go:122] "RemoveContainer" containerID="6c6901b6363666f790a0e8abf204b6ac0aeba6a1d7937b341eaec72d342ee270" Nov 02 14:38:46.230442 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:46.230425 4866 scope.go:122] "RemoveContainer" containerID="6c6901b6363666f790a0e8abf204b6ac0aeba6a1d7937b341eaec72d342ee270" Nov 02 14:38:46.230738 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:38:46.230717 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6901b6363666f790a0e8abf204b6ac0aeba6a1d7937b341eaec72d342ee270\": container with ID starting with 6c6901b6363666f790a0e8abf204b6ac0aeba6a1d7937b341eaec72d342ee270 not found: ID does not exist" containerID="6c6901b6363666f790a0e8abf204b6ac0aeba6a1d7937b341eaec72d342ee270" Nov 02 14:38:46.230790 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:46.230748 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6901b6363666f790a0e8abf204b6ac0aeba6a1d7937b341eaec72d342ee270"} err="failed to get container status \"6c6901b6363666f790a0e8abf204b6ac0aeba6a1d7937b341eaec72d342ee270\": rpc error: code = NotFound desc = could not find container \"6c6901b6363666f790a0e8abf204b6ac0aeba6a1d7937b341eaec72d342ee270\": container with ID starting with 6c6901b6363666f790a0e8abf204b6ac0aeba6a1d7937b341eaec72d342ee270 not found: ID does not exist" Nov 02 14:38:46.242788 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:38:46.242754 4866 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7813-09c805\" is forbidden: User \"system:node:ip-10-0-134-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-99.ec2.internal' and this object" podUID="e58e566f-2681-4b33-879e-8ed9331da382" pod="opendatahub/test-curl-7813-09c805" Nov 02 14:38:46.946470 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:46.946434 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e58e566f-2681-4b33-879e-8ed9331da382" path="/var/lib/kubelet/pods/e58e566f-2681-4b33-879e-8ed9331da382/volumes" Nov 02 14:38:47.435910 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:47.435885 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7813-b1ec9c" Nov 02 14:38:47.537717 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:47.537629 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39907e49-1f0d-4a5b-a4b3-f44d93564ed2-kube-api-access-zql2k\" (UniqueName: \"kubernetes.io/projected/39907e49-1f0d-4a5b-a4b3-f44d93564ed2-kube-api-access-zql2k\") pod \"39907e49-1f0d-4a5b-a4b3-f44d93564ed2\" (UID: \"39907e49-1f0d-4a5b-a4b3-f44d93564ed2\") " Nov 02 14:38:47.539759 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:47.539730 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39907e49-1f0d-4a5b-a4b3-f44d93564ed2-kube-api-access-zql2k" pod "39907e49-1f0d-4a5b-a4b3-f44d93564ed2" (UID: "39907e49-1f0d-4a5b-a4b3-f44d93564ed2"). InnerVolumeSpecName "kube-api-access-zql2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:38:47.639367 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:47.639326 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zql2k\" (UniqueName: \"kubernetes.io/projected/39907e49-1f0d-4a5b-a4b3-f44d93564ed2-kube-api-access-zql2k\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:38:48.234772 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:48.234733 4866 generic.go:356] "Generic (PLEG): container finished" podID="39907e49-1f0d-4a5b-a4b3-f44d93564ed2" containerID="22c1ceadb678f74180f42907b5cd796f5422184de6cb480e80b1b120f7eb07c8" exitCode=137 Nov 02 14:38:48.235020 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:48.234779 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7813-b1ec9c" Nov 02 14:38:48.235020 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:48.234833 4866 scope.go:122] "RemoveContainer" containerID="22c1ceadb678f74180f42907b5cd796f5422184de6cb480e80b1b120f7eb07c8" Nov 02 14:38:48.243932 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:48.243913 4866 scope.go:122] "RemoveContainer" containerID="22c1ceadb678f74180f42907b5cd796f5422184de6cb480e80b1b120f7eb07c8" Nov 02 14:38:48.244356 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:38:48.244337 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c1ceadb678f74180f42907b5cd796f5422184de6cb480e80b1b120f7eb07c8\": container with ID starting with 22c1ceadb678f74180f42907b5cd796f5422184de6cb480e80b1b120f7eb07c8 not found: ID does not exist" containerID="22c1ceadb678f74180f42907b5cd796f5422184de6cb480e80b1b120f7eb07c8" Nov 02 14:38:48.244396 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:48.244366 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c1ceadb678f74180f42907b5cd796f5422184de6cb480e80b1b120f7eb07c8"} err="failed to get container status \"22c1ceadb678f74180f42907b5cd796f5422184de6cb480e80b1b120f7eb07c8\": rpc error: code = NotFound desc = could not find container \"22c1ceadb678f74180f42907b5cd796f5422184de6cb480e80b1b120f7eb07c8\": container with ID starting with 22c1ceadb678f74180f42907b5cd796f5422184de6cb480e80b1b120f7eb07c8 not found: ID does not exist" Nov 02 14:38:48.257000 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:38:48.256969 4866 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7813-b1ec9c\" is forbidden: User \"system:node:ip-10-0-134-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-99.ec2.internal' and this object" podUID="39907e49-1f0d-4a5b-a4b3-f44d93564ed2" pod="opendatahub/test-curl-7813-b1ec9c" Nov 02 14:38:48.950756 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:48.950711 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39907e49-1f0d-4a5b-a4b3-f44d93564ed2" path="/var/lib/kubelet/pods/39907e49-1f0d-4a5b-a4b3-f44d93564ed2/volumes" Nov 02 14:38:48.954868 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:38:48.954839 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x"] Nov 02 14:39:03.975473 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:03.975413 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" podUID="df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" containerName="main" containerID="cri-o://81b79ec20f6323afff1689b8a8eeb98e68ca9c46dd7a5de411398d1864514bbc" gracePeriod=45 Nov 02 14:39:04.246651 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.246623 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:39:04.268793 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.268755 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-model-cache\") pod \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " Nov 02 14:39:04.268977 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.268806 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-kube-api-access-99j74\" (UniqueName: \"kubernetes.io/projected/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-kube-api-access-99j74\") pod \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " Nov 02 14:39:04.268977 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.268831 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-dshm\") pod \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " Nov 02 14:39:04.268977 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.268866 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-tls-certs\") pod \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " Nov 02 14:39:04.268977 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.268902 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-tmp-dir\") pod \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " Nov 02 14:39:04.269214 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.269024 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-home\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-home\") pod \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\" (UID: \"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb\") " Nov 02 14:39:04.269214 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.269125 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-model-cache" pod "df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" (UID: "df7e11b5-e20c-4a71-b53b-1eb6b2de67fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:04.269309 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.269289 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-tmp-dir" pod "df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" (UID: "df7e11b5-e20c-4a71-b53b-1eb6b2de67fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:04.269360 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.269308 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-home" pod "df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" (UID: "df7e11b5-e20c-4a71-b53b-1eb6b2de67fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:04.269495 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.269477 4866 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-tmp-dir\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:04.269532 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.269506 4866 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-home\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:04.269532 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.269516 4866 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-model-cache\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:04.271880 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.271847 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-tls-certs" pod "df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" (UID: "df7e11b5-e20c-4a71-b53b-1eb6b2de67fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:39:04.272030 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.271996 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-kube-api-access-99j74" pod "df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" (UID: "df7e11b5-e20c-4a71-b53b-1eb6b2de67fb"). InnerVolumeSpecName "kube-api-access-99j74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:39:04.272097 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.272049 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-dshm" pod "df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" (UID: "df7e11b5-e20c-4a71-b53b-1eb6b2de67fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:04.355428 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.355391 4866 generic.go:356] "Generic (PLEG): container finished" podID="df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" containerID="81b79ec20f6323afff1689b8a8eeb98e68ca9c46dd7a5de411398d1864514bbc" exitCode=0 Nov 02 14:39:04.355612 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.355438 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" Nov 02 14:39:04.355612 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.355471 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" event={"ID":"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb","Type":"ContainerDied","Data":"81b79ec20f6323afff1689b8a8eeb98e68ca9c46dd7a5de411398d1864514bbc"} Nov 02 14:39:04.355612 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.355515 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x" event={"ID":"df7e11b5-e20c-4a71-b53b-1eb6b2de67fb","Type":"ContainerDied","Data":"19ffdce1255d1cf0aa47df9ff059c6b29e68b919512872bde587ad58f0c59359"} Nov 02 14:39:04.355612 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.355534 4866 scope.go:122] "RemoveContainer" containerID="81b79ec20f6323afff1689b8a8eeb98e68ca9c46dd7a5de411398d1864514bbc" Nov 02 14:39:04.366999 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.366925 4866 scope.go:122] "RemoveContainer" containerID="81b79ec20f6323afff1689b8a8eeb98e68ca9c46dd7a5de411398d1864514bbc" Nov 02 14:39:04.367323 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:39:04.367304 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b79ec20f6323afff1689b8a8eeb98e68ca9c46dd7a5de411398d1864514bbc\": container with ID starting with 81b79ec20f6323afff1689b8a8eeb98e68ca9c46dd7a5de411398d1864514bbc not found: ID does not exist" containerID="81b79ec20f6323afff1689b8a8eeb98e68ca9c46dd7a5de411398d1864514bbc" Nov 02 14:39:04.367373 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.367334 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b79ec20f6323afff1689b8a8eeb98e68ca9c46dd7a5de411398d1864514bbc"} err="failed to get container status \"81b79ec20f6323afff1689b8a8eeb98e68ca9c46dd7a5de411398d1864514bbc\": rpc error: code = NotFound desc = could not find container \"81b79ec20f6323afff1689b8a8eeb98e68ca9c46dd7a5de411398d1864514bbc\": container with ID starting with 81b79ec20f6323afff1689b8a8eeb98e68ca9c46dd7a5de411398d1864514bbc not found: ID does not exist" Nov 02 14:39:04.370546 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.370517 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99j74\" (UniqueName: \"kubernetes.io/projected/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-kube-api-access-99j74\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:04.370656 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.370550 4866 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-dshm\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:04.370656 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.370566 4866 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb-tls-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:04.411429 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.411393 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x"] Nov 02 14:39:04.415635 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.415603 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-555d1e/premium-w0-555d1e-kserve-79486946df-n8m9x"] Nov 02 14:39:04.945464 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:04.945427 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" path="/var/lib/kubelet/pods/df7e11b5-e20c-4a71-b53b-1eb6b2de67fb/volumes" Nov 02 14:39:06.012533 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:39:06.012493 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:39:06.996163 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:06.996129 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g"] Nov 02 14:39:11.997114 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:11.997063 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" podUID="e7cb0799-8ad5-431f-850d-fbfaa881f9e5" containerName="payload-processing" containerID="cri-o://4836afc1eaaf69f6c3f959d78019024f1ef31b9a7d774ad54b69ce9fc4184313" gracePeriod=25 Nov 02 14:39:12.949597 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:12.949556 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" podUID="e7cb0799-8ad5-431f-850d-fbfaa881f9e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:39:22.068506 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.068464 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh"] Nov 02 14:39:22.069050 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.068729 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" podUID="b1725bed-2118-40d7-bfa7-88cf6ca87354" containerName="maas-api" containerID="cri-o://ce02e2cd0972810846e4211d758320f49a0a2105618645e7517501723319204e" gracePeriod=30 Nov 02 14:39:22.334695 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.334622 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" Nov 02 14:39:22.463994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.463941 4866 generic.go:356] "Generic (PLEG): container finished" podID="b1725bed-2118-40d7-bfa7-88cf6ca87354" containerID="ce02e2cd0972810846e4211d758320f49a0a2105618645e7517501723319204e" exitCode=0 Nov 02 14:39:22.463994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.463977 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" event={"ID":"b1725bed-2118-40d7-bfa7-88cf6ca87354","Type":"ContainerDied","Data":"ce02e2cd0972810846e4211d758320f49a0a2105618645e7517501723319204e"} Nov 02 14:39:22.464227 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.464026 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" Nov 02 14:39:22.464227 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.464037 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh" event={"ID":"b1725bed-2118-40d7-bfa7-88cf6ca87354","Type":"ContainerDied","Data":"e6ce0582520cdea96b9c3cebe31009feee18f941fae31570d6d568c2822dc258"} Nov 02 14:39:22.464227 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.464062 4866 scope.go:122] "RemoveContainer" containerID="ce02e2cd0972810846e4211d758320f49a0a2105618645e7517501723319204e" Nov 02 14:39:22.473583 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.473558 4866 scope.go:122] "RemoveContainer" containerID="ce02e2cd0972810846e4211d758320f49a0a2105618645e7517501723319204e" Nov 02 14:39:22.473866 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:39:22.473845 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce02e2cd0972810846e4211d758320f49a0a2105618645e7517501723319204e\": container with ID starting with ce02e2cd0972810846e4211d758320f49a0a2105618645e7517501723319204e not found: ID does not exist" containerID="ce02e2cd0972810846e4211d758320f49a0a2105618645e7517501723319204e" Nov 02 14:39:22.473936 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.473878 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce02e2cd0972810846e4211d758320f49a0a2105618645e7517501723319204e"} err="failed to get container status \"ce02e2cd0972810846e4211d758320f49a0a2105618645e7517501723319204e\": rpc error: code = NotFound desc = could not find container \"ce02e2cd0972810846e4211d758320f49a0a2105618645e7517501723319204e\": container with ID starting with ce02e2cd0972810846e4211d758320f49a0a2105618645e7517501723319204e not found: ID does not exist" Nov 02 14:39:22.480006 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.479978 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1725bed-2118-40d7-bfa7-88cf6ca87354-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1725bed-2118-40d7-bfa7-88cf6ca87354-metrics-certs\") pod \"b1725bed-2118-40d7-bfa7-88cf6ca87354\" (UID: \"b1725bed-2118-40d7-bfa7-88cf6ca87354\") " Nov 02 14:39:22.480130 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.480066 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1725bed-2118-40d7-bfa7-88cf6ca87354-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1725bed-2118-40d7-bfa7-88cf6ca87354-openshift-service-ca\") pod \"b1725bed-2118-40d7-bfa7-88cf6ca87354\" (UID: \"b1725bed-2118-40d7-bfa7-88cf6ca87354\") " Nov 02 14:39:22.480130 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.480122 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1725bed-2118-40d7-bfa7-88cf6ca87354-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1725bed-2118-40d7-bfa7-88cf6ca87354-maas-api-tls\") pod \"b1725bed-2118-40d7-bfa7-88cf6ca87354\" (UID: \"b1725bed-2118-40d7-bfa7-88cf6ca87354\") " Nov 02 14:39:22.480259 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.480177 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1725bed-2118-40d7-bfa7-88cf6ca87354-kube-api-access-jb76q\" (UniqueName: \"kubernetes.io/projected/b1725bed-2118-40d7-bfa7-88cf6ca87354-kube-api-access-jb76q\") pod \"b1725bed-2118-40d7-bfa7-88cf6ca87354\" (UID: \"b1725bed-2118-40d7-bfa7-88cf6ca87354\") " Nov 02 14:39:22.480455 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.480427 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1725bed-2118-40d7-bfa7-88cf6ca87354-openshift-service-ca" pod "b1725bed-2118-40d7-bfa7-88cf6ca87354" (UID: "b1725bed-2118-40d7-bfa7-88cf6ca87354"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:39:22.480626 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.480609 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1725bed-2118-40d7-bfa7-88cf6ca87354-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:22.482241 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.482212 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1725bed-2118-40d7-bfa7-88cf6ca87354-metrics-certs" pod "b1725bed-2118-40d7-bfa7-88cf6ca87354" (UID: "b1725bed-2118-40d7-bfa7-88cf6ca87354"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:39:22.482332 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.482314 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1725bed-2118-40d7-bfa7-88cf6ca87354-maas-api-tls" pod "b1725bed-2118-40d7-bfa7-88cf6ca87354" (UID: "b1725bed-2118-40d7-bfa7-88cf6ca87354"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:39:22.482566 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.482543 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1725bed-2118-40d7-bfa7-88cf6ca87354-kube-api-access-jb76q" pod "b1725bed-2118-40d7-bfa7-88cf6ca87354" (UID: "b1725bed-2118-40d7-bfa7-88cf6ca87354"). InnerVolumeSpecName "kube-api-access-jb76q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:39:22.581623 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.581586 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1725bed-2118-40d7-bfa7-88cf6ca87354-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:22.581623 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.581617 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1725bed-2118-40d7-bfa7-88cf6ca87354-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:22.581623 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.581627 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb76q\" (UniqueName: \"kubernetes.io/projected/b1725bed-2118-40d7-bfa7-88cf6ca87354-kube-api-access-jb76q\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:22.812501 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.812461 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh"] Nov 02 14:39:22.816191 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.816165 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-555d1e-777689d879-ddcbh"] Nov 02 14:39:22.948124 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.948089 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1725bed-2118-40d7-bfa7-88cf6ca87354" path="/var/lib/kubelet/pods/b1725bed-2118-40d7-bfa7-88cf6ca87354/volumes" Nov 02 14:39:22.949609 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:22.949579 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" podUID="e7cb0799-8ad5-431f-850d-fbfaa881f9e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:39:32.255274 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:32.255208 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" Nov 02 14:39:32.396995 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:32.396939 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7cb0799-8ad5-431f-850d-fbfaa881f9e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7cb0799-8ad5-431f-850d-fbfaa881f9e5-plugins-config-volume\") pod \"e7cb0799-8ad5-431f-850d-fbfaa881f9e5\" (UID: \"e7cb0799-8ad5-431f-850d-fbfaa881f9e5\") " Nov 02 14:39:32.397188 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:32.397090 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7cb0799-8ad5-431f-850d-fbfaa881f9e5-kube-api-access-px9p5\" (UniqueName: \"kubernetes.io/projected/e7cb0799-8ad5-431f-850d-fbfaa881f9e5-kube-api-access-px9p5\") pod \"e7cb0799-8ad5-431f-850d-fbfaa881f9e5\" (UID: \"e7cb0799-8ad5-431f-850d-fbfaa881f9e5\") " Nov 02 14:39:32.397380 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:32.397354 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7cb0799-8ad5-431f-850d-fbfaa881f9e5-plugins-config-volume" pod "e7cb0799-8ad5-431f-850d-fbfaa881f9e5" (UID: "e7cb0799-8ad5-431f-850d-fbfaa881f9e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:39:32.399342 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:32.399309 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7cb0799-8ad5-431f-850d-fbfaa881f9e5-kube-api-access-px9p5" pod "e7cb0799-8ad5-431f-850d-fbfaa881f9e5" (UID: "e7cb0799-8ad5-431f-850d-fbfaa881f9e5"). InnerVolumeSpecName "kube-api-access-px9p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:39:32.498561 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:32.498520 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px9p5\" (UniqueName: \"kubernetes.io/projected/e7cb0799-8ad5-431f-850d-fbfaa881f9e5-kube-api-access-px9p5\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:32.498561 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:32.498550 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7cb0799-8ad5-431f-850d-fbfaa881f9e5-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:32.523239 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:32.523149 4866 generic.go:356] "Generic (PLEG): container finished" podID="e7cb0799-8ad5-431f-850d-fbfaa881f9e5" containerID="4836afc1eaaf69f6c3f959d78019024f1ef31b9a7d774ad54b69ce9fc4184313" exitCode=0 Nov 02 14:39:32.523239 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:32.523197 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" Nov 02 14:39:32.523239 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:32.523231 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" event={"ID":"e7cb0799-8ad5-431f-850d-fbfaa881f9e5","Type":"ContainerDied","Data":"4836afc1eaaf69f6c3f959d78019024f1ef31b9a7d774ad54b69ce9fc4184313"} Nov 02 14:39:32.523485 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:32.523284 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g" event={"ID":"e7cb0799-8ad5-431f-850d-fbfaa881f9e5","Type":"ContainerDied","Data":"a810dc9a134e422548c9d80dea1ecb6fb78739140e01b993e27adfd09b986c80"} Nov 02 14:39:32.523485 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:32.523312 4866 scope.go:122] "RemoveContainer" containerID="4836afc1eaaf69f6c3f959d78019024f1ef31b9a7d774ad54b69ce9fc4184313" Nov 02 14:39:32.533322 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:32.533301 4866 scope.go:122] "RemoveContainer" containerID="4836afc1eaaf69f6c3f959d78019024f1ef31b9a7d774ad54b69ce9fc4184313" Nov 02 14:39:32.533591 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:39:32.533565 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4836afc1eaaf69f6c3f959d78019024f1ef31b9a7d774ad54b69ce9fc4184313\": container with ID starting with 4836afc1eaaf69f6c3f959d78019024f1ef31b9a7d774ad54b69ce9fc4184313 not found: ID does not exist" containerID="4836afc1eaaf69f6c3f959d78019024f1ef31b9a7d774ad54b69ce9fc4184313" Nov 02 14:39:32.533692 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:32.533598 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4836afc1eaaf69f6c3f959d78019024f1ef31b9a7d774ad54b69ce9fc4184313"} err="failed to get container status \"4836afc1eaaf69f6c3f959d78019024f1ef31b9a7d774ad54b69ce9fc4184313\": rpc error: code = NotFound desc = could not find container \"4836afc1eaaf69f6c3f959d78019024f1ef31b9a7d774ad54b69ce9fc4184313\": container with ID starting with 4836afc1eaaf69f6c3f959d78019024f1ef31b9a7d774ad54b69ce9fc4184313 not found: ID does not exist" Nov 02 14:39:32.578724 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:32.578690 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g"] Nov 02 14:39:32.590118 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:32.590091 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-555d1e-758c9888f-6hs9g"] Nov 02 14:39:32.945293 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:32.945264 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7cb0799-8ad5-431f-850d-fbfaa881f9e5" path="/var/lib/kubelet/pods/e7cb0799-8ad5-431f-850d-fbfaa881f9e5/volumes" Nov 02 14:39:34.341974 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:34.341922 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4"] Nov 02 14:39:35.807640 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:35.807607 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6"] Nov 02 14:39:35.825472 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:35.825435 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m"] Nov 02 14:39:36.010503 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:39:36.010469 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:39:38.244335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:38.244294 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth"] Nov 02 14:39:49.363264 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.363212 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" podUID="844d3fe8-1341-4a06-82f0-f119401c4a6c" containerName="main" containerID="cri-o://13503b7ba366e7a57736d4792dff24adb26c33072a47f4bf4c5f0a978ae8c6f2" gracePeriod=45 Nov 02 14:39:49.622084 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.622052 4866 generic.go:356] "Generic (PLEG): container finished" podID="844d3fe8-1341-4a06-82f0-f119401c4a6c" containerID="13503b7ba366e7a57736d4792dff24adb26c33072a47f4bf4c5f0a978ae8c6f2" exitCode=0 Nov 02 14:39:49.622244 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.622109 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" event={"ID":"844d3fe8-1341-4a06-82f0-f119401c4a6c","Type":"ContainerDied","Data":"13503b7ba366e7a57736d4792dff24adb26c33072a47f4bf4c5f0a978ae8c6f2"} Nov 02 14:39:49.622244 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.622152 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" event={"ID":"844d3fe8-1341-4a06-82f0-f119401c4a6c","Type":"ContainerDied","Data":"8af73f32dc3339985c3612ebc99cc3c8833998445ffe6cbecbf45dfeee5bc62d"} Nov 02 14:39:49.622244 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.622167 4866 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8af73f32dc3339985c3612ebc99cc3c8833998445ffe6cbecbf45dfeee5bc62d" Nov 02 14:39:49.632449 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.632424 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:39:49.793226 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.793186 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-tmp-dir\") pod \"844d3fe8-1341-4a06-82f0-f119401c4a6c\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " Nov 02 14:39:49.793429 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.793242 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-dshm\") pod \"844d3fe8-1341-4a06-82f0-f119401c4a6c\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " Nov 02 14:39:49.793429 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.793284 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-model-cache\") pod \"844d3fe8-1341-4a06-82f0-f119401c4a6c\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " Nov 02 14:39:49.793429 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.793305 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/844d3fe8-1341-4a06-82f0-f119401c4a6c-tls-certs\" (UniqueName: \"kubernetes.io/secret/844d3fe8-1341-4a06-82f0-f119401c4a6c-tls-certs\") pod \"844d3fe8-1341-4a06-82f0-f119401c4a6c\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " Nov 02 14:39:49.793429 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.793357 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-home\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-home\") pod \"844d3fe8-1341-4a06-82f0-f119401c4a6c\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " Nov 02 14:39:49.793429 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.793402 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844d3fe8-1341-4a06-82f0-f119401c4a6c-kube-api-access-cjn8x\" (UniqueName: \"kubernetes.io/projected/844d3fe8-1341-4a06-82f0-f119401c4a6c-kube-api-access-cjn8x\") pod \"844d3fe8-1341-4a06-82f0-f119401c4a6c\" (UID: \"844d3fe8-1341-4a06-82f0-f119401c4a6c\") " Nov 02 14:39:49.793697 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.793553 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-tmp-dir" pod "844d3fe8-1341-4a06-82f0-f119401c4a6c" (UID: "844d3fe8-1341-4a06-82f0-f119401c4a6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:49.793697 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.793589 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-model-cache" pod "844d3fe8-1341-4a06-82f0-f119401c4a6c" (UID: "844d3fe8-1341-4a06-82f0-f119401c4a6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:49.793777 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.793707 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-home" pod "844d3fe8-1341-4a06-82f0-f119401c4a6c" (UID: "844d3fe8-1341-4a06-82f0-f119401c4a6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:49.793885 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.793864 4866 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-model-cache\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:49.793942 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.793887 4866 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-home\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:49.793942 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.793895 4866 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-tmp-dir\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:49.795599 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.795576 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-dshm" pod "844d3fe8-1341-4a06-82f0-f119401c4a6c" (UID: "844d3fe8-1341-4a06-82f0-f119401c4a6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:49.795694 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.795679 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/844d3fe8-1341-4a06-82f0-f119401c4a6c-tls-certs" pod "844d3fe8-1341-4a06-82f0-f119401c4a6c" (UID: "844d3fe8-1341-4a06-82f0-f119401c4a6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:39:49.795756 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.795721 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844d3fe8-1341-4a06-82f0-f119401c4a6c-kube-api-access-cjn8x" pod "844d3fe8-1341-4a06-82f0-f119401c4a6c" (UID: "844d3fe8-1341-4a06-82f0-f119401c4a6c"). InnerVolumeSpecName "kube-api-access-cjn8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:39:49.895520 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.895425 4866 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/844d3fe8-1341-4a06-82f0-f119401c4a6c-dshm\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:49.895520 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.895471 4866 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/844d3fe8-1341-4a06-82f0-f119401c4a6c-tls-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:49.895520 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:49.895488 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjn8x\" (UniqueName: \"kubernetes.io/projected/844d3fe8-1341-4a06-82f0-f119401c4a6c-kube-api-access-cjn8x\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:50.022261 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.022200 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh"] Nov 02 14:39:50.022936 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.022917 4866 cpu_manager.go:414] "containerMap: removing container" podUID="df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" containerName="main" Nov 02 14:39:50.023038 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.022936 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" containerName="main" Nov 02 14:39:50.023038 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.022968 4866 cpu_manager.go:414] "containerMap: removing container" podUID="844d3fe8-1341-4a06-82f0-f119401c4a6c" containerName="main" Nov 02 14:39:50.023038 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.022974 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844d3fe8-1341-4a06-82f0-f119401c4a6c" containerName="main" Nov 02 14:39:50.023038 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.022986 4866 cpu_manager.go:414] "containerMap: removing container" podUID="b1725bed-2118-40d7-bfa7-88cf6ca87354" containerName="maas-api" Nov 02 14:39:50.023038 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.022992 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1725bed-2118-40d7-bfa7-88cf6ca87354" containerName="maas-api" Nov 02 14:39:50.023038 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.023006 4866 cpu_manager.go:414] "containerMap: removing container" podUID="e7cb0799-8ad5-431f-850d-fbfaa881f9e5" containerName="payload-processing" Nov 02 14:39:50.023038 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.023011 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7cb0799-8ad5-431f-850d-fbfaa881f9e5" containerName="payload-processing" Nov 02 14:39:50.023038 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.023029 4866 cpu_manager.go:414] "containerMap: removing container" podUID="39907e49-1f0d-4a5b-a4b3-f44d93564ed2" containerName="test-curl-7813-b1ec9c" Nov 02 14:39:50.023038 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.023033 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39907e49-1f0d-4a5b-a4b3-f44d93564ed2" containerName="test-curl-7813-b1ec9c" Nov 02 14:39:50.023491 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.023108 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="39907e49-1f0d-4a5b-a4b3-f44d93564ed2" containerName="test-curl-7813-b1ec9c" Nov 02 14:39:50.023491 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.023119 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="df7e11b5-e20c-4a71-b53b-1eb6b2de67fb" containerName="main" Nov 02 14:39:50.023491 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.023127 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="844d3fe8-1341-4a06-82f0-f119401c4a6c" containerName="main" Nov 02 14:39:50.023491 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.023133 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1725bed-2118-40d7-bfa7-88cf6ca87354" containerName="maas-api" Nov 02 14:39:50.023491 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.023141 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7cb0799-8ad5-431f-850d-fbfaa881f9e5" containerName="payload-processing" Nov 02 14:39:50.026727 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.026705 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.034276 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.034245 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh"] Nov 02 14:39:50.043271 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.043231 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-866159\"/\"kube-root-ca.crt\"" Nov 02 14:39:50.043410 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.043348 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-866159\"/\"openshift-service-ca.crt\"" Nov 02 14:39:50.043482 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.043470 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-866159\"/\"default-dockercfg-9x58g\"" Nov 02 14:39:50.043534 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.043489 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-866159\"/\"e2e-conflict-a-be5aa686-kserve-self-signed-certs\"" Nov 02 14:39:50.098281 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.098243 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-dshm\") pod \"e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.098479 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.098304 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-model-cache\") pod \"e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.098549 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.098469 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-tmp-dir\") pod \"e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.098549 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.098512 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msrb7\" (UniqueName: \"kubernetes.io/projected/29dea1dd-e792-415a-9c82-dfa2edf71510-kube-api-access-msrb7\") pod \"e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.098647 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.098562 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-home\") pod \"e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.098647 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.098590 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29dea1dd-e792-415a-9c82-dfa2edf71510-tls-certs\") pod \"e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.201257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.200478 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-dshm\") pod \"e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.201257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.200578 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-model-cache\") pod \"e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.201257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.200695 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-tmp-dir\") pod \"e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.201257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.200739 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msrb7\" (UniqueName: \"kubernetes.io/projected/29dea1dd-e792-415a-9c82-dfa2edf71510-kube-api-access-msrb7\") pod \"e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.201257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.200798 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-home\") pod \"e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.201257 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.200820 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29dea1dd-e792-415a-9c82-dfa2edf71510-tls-certs\") pod \"e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.202804 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.201487 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-tmp-dir\") pod \"e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.202804 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.201857 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-model-cache\") pod \"e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.202804 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.202453 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-home\") pod \"e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.206301 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.206273 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-dshm\") pod \"e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.207740 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.207688 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29dea1dd-e792-415a-9c82-dfa2edf71510-tls-certs\") pod \"e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.226711 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.226664 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msrb7\" (UniqueName: \"kubernetes.io/projected/29dea1dd-e792-415a-9c82-dfa2edf71510-kube-api-access-msrb7\") pod \"e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.339837 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.339798 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:50.627317 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.627236 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4" Nov 02 14:39:50.643205 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.643179 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh"] Nov 02 14:39:50.645087 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:39:50.645058 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29dea1dd_e792_415a_9c82_dfa2edf71510.slice/crio-6f410d52fde91b5522b3c8c77eea3315ad95ef838efbb5cdfc543d69f9c23f13 WatchSource:0}: Error finding container 6f410d52fde91b5522b3c8c77eea3315ad95ef838efbb5cdfc543d69f9c23f13: Status 404 returned error can't find the container with id 6f410d52fde91b5522b3c8c77eea3315ad95ef838efbb5cdfc543d69f9c23f13 Nov 02 14:39:50.679222 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.679194 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4"] Nov 02 14:39:50.683510 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.683485 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e07583/sub-test-model-e07583-kserve-58568f678f-zmnd4"] Nov 02 14:39:50.828070 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.828028 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" podUID="d7037eab-3f5f-419b-94f6-784ff7331925" containerName="main" containerID="cri-o://10be2c01de270a3c93942abe29b60816d4edbe45088b247941c692cd1cf1c579" gracePeriod=45 Nov 02 14:39:50.832696 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.832666 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" podUID="d7037eab-3f5f-419b-94f6-784ff7331925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": EOF" Nov 02 14:39:50.845504 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.845466 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" podUID="9753ddaa-3899-46ea-952f-5f03d76037c6" containerName="main" containerID="cri-o://e3f2beeb5ba2f2b69cee8f37a17604b46aba36ab4ba45c78a293f2fc46204c4e" gracePeriod=45 Nov 02 14:39:50.948356 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:50.948326 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844d3fe8-1341-4a06-82f0-f119401c4a6c" path="/var/lib/kubelet/pods/844d3fe8-1341-4a06-82f0-f119401c4a6c/volumes" Nov 02 14:39:51.085777 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.085747 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:39:51.120305 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.120274 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:39:51.215567 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.215530 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-home\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-home\") pod \"d7037eab-3f5f-419b-94f6-784ff7331925\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " Nov 02 14:39:51.215773 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.215591 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-model-cache\") pod \"9753ddaa-3899-46ea-952f-5f03d76037c6\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " Nov 02 14:39:51.215773 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.215624 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9753ddaa-3899-46ea-952f-5f03d76037c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/9753ddaa-3899-46ea-952f-5f03d76037c6-tls-certs\") pod \"9753ddaa-3899-46ea-952f-5f03d76037c6\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " Nov 02 14:39:51.215773 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.215652 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-dshm\") pod \"9753ddaa-3899-46ea-952f-5f03d76037c6\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " Nov 02 14:39:51.215773 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.215687 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7037eab-3f5f-419b-94f6-784ff7331925-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7037eab-3f5f-419b-94f6-784ff7331925-tls-certs\") pod \"d7037eab-3f5f-419b-94f6-784ff7331925\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " Nov 02 14:39:51.215773 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.215721 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9753ddaa-3899-46ea-952f-5f03d76037c6-kube-api-access-bdn8d\" (UniqueName: \"kubernetes.io/projected/9753ddaa-3899-46ea-952f-5f03d76037c6-kube-api-access-bdn8d\") pod \"9753ddaa-3899-46ea-952f-5f03d76037c6\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " Nov 02 14:39:51.215773 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.215767 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7037eab-3f5f-419b-94f6-784ff7331925-kube-api-access-cq4zn\" (UniqueName: \"kubernetes.io/projected/d7037eab-3f5f-419b-94f6-784ff7331925-kube-api-access-cq4zn\") pod \"d7037eab-3f5f-419b-94f6-784ff7331925\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " Nov 02 14:39:51.216094 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.215831 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-tmp-dir\") pod \"9753ddaa-3899-46ea-952f-5f03d76037c6\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " Nov 02 14:39:51.216094 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.215855 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-home\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-home\") pod \"9753ddaa-3899-46ea-952f-5f03d76037c6\" (UID: \"9753ddaa-3899-46ea-952f-5f03d76037c6\") " Nov 02 14:39:51.216094 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.215885 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-home" pod "d7037eab-3f5f-419b-94f6-784ff7331925" (UID: "d7037eab-3f5f-419b-94f6-784ff7331925"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:51.216264 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.216147 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-home" pod "9753ddaa-3899-46ea-952f-5f03d76037c6" (UID: "9753ddaa-3899-46ea-952f-5f03d76037c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:51.216264 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.216160 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-tmp-dir" pod "9753ddaa-3899-46ea-952f-5f03d76037c6" (UID: "9753ddaa-3899-46ea-952f-5f03d76037c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:51.216377 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.216269 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-dshm\") pod \"d7037eab-3f5f-419b-94f6-784ff7331925\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " Nov 02 14:39:51.216377 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.216340 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-model-cache" pod "9753ddaa-3899-46ea-952f-5f03d76037c6" (UID: "9753ddaa-3899-46ea-952f-5f03d76037c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:51.216480 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.216428 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-tmp-dir\") pod \"d7037eab-3f5f-419b-94f6-784ff7331925\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " Nov 02 14:39:51.216480 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.216461 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-model-cache\") pod \"d7037eab-3f5f-419b-94f6-784ff7331925\" (UID: \"d7037eab-3f5f-419b-94f6-784ff7331925\") " Nov 02 14:39:51.216843 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.216814 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-model-cache" pod "d7037eab-3f5f-419b-94f6-784ff7331925" (UID: "d7037eab-3f5f-419b-94f6-784ff7331925"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:51.217088 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.217061 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-tmp-dir" pod "d7037eab-3f5f-419b-94f6-784ff7331925" (UID: "d7037eab-3f5f-419b-94f6-784ff7331925"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:51.217196 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.217071 4866 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-tmp-dir\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.217196 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.217113 4866 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-home\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.217196 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.217129 4866 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-model-cache\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.217196 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.217143 4866 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-home\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.217196 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.217157 4866 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-model-cache\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.218745 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.218713 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9753ddaa-3899-46ea-952f-5f03d76037c6-tls-certs" pod "9753ddaa-3899-46ea-952f-5f03d76037c6" (UID: "9753ddaa-3899-46ea-952f-5f03d76037c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:39:51.218922 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.218848 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9753ddaa-3899-46ea-952f-5f03d76037c6-kube-api-access-bdn8d" pod "9753ddaa-3899-46ea-952f-5f03d76037c6" (UID: "9753ddaa-3899-46ea-952f-5f03d76037c6"). InnerVolumeSpecName "kube-api-access-bdn8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:39:51.219424 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.219396 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-dshm" pod "d7037eab-3f5f-419b-94f6-784ff7331925" (UID: "d7037eab-3f5f-419b-94f6-784ff7331925"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:51.219515 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.219424 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-dshm" pod "9753ddaa-3899-46ea-952f-5f03d76037c6" (UID: "9753ddaa-3899-46ea-952f-5f03d76037c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:51.219515 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.219403 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7037eab-3f5f-419b-94f6-784ff7331925-tls-certs" pod "d7037eab-3f5f-419b-94f6-784ff7331925" (UID: "d7037eab-3f5f-419b-94f6-784ff7331925"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:39:51.220028 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.220008 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7037eab-3f5f-419b-94f6-784ff7331925-kube-api-access-cq4zn" pod "d7037eab-3f5f-419b-94f6-784ff7331925" (UID: "d7037eab-3f5f-419b-94f6-784ff7331925"). InnerVolumeSpecName "kube-api-access-cq4zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:39:51.318607 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.318568 4866 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-tmp-dir\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.318607 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.318599 4866 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9753ddaa-3899-46ea-952f-5f03d76037c6-tls-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.318607 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.318612 4866 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9753ddaa-3899-46ea-952f-5f03d76037c6-dshm\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.318846 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.318625 4866 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7037eab-3f5f-419b-94f6-784ff7331925-tls-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.318846 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.318634 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdn8d\" (UniqueName: \"kubernetes.io/projected/9753ddaa-3899-46ea-952f-5f03d76037c6-kube-api-access-bdn8d\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.318846 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.318644 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq4zn\" (UniqueName: \"kubernetes.io/projected/d7037eab-3f5f-419b-94f6-784ff7331925-kube-api-access-cq4zn\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.318846 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.318652 4866 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7037eab-3f5f-419b-94f6-784ff7331925-dshm\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.633892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.633803 4866 generic.go:356] "Generic (PLEG): container finished" podID="d7037eab-3f5f-419b-94f6-784ff7331925" containerID="10be2c01de270a3c93942abe29b60816d4edbe45088b247941c692cd1cf1c579" exitCode=0 Nov 02 14:39:51.633892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.633852 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" Nov 02 14:39:51.634424 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.633888 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" event={"ID":"d7037eab-3f5f-419b-94f6-784ff7331925","Type":"ContainerDied","Data":"10be2c01de270a3c93942abe29b60816d4edbe45088b247941c692cd1cf1c579"} Nov 02 14:39:51.634424 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.633939 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6" event={"ID":"d7037eab-3f5f-419b-94f6-784ff7331925","Type":"ContainerDied","Data":"abf571d0699e7db6acd193779ff250e0419659c3538c948c7d5955d9d6f310fc"} Nov 02 14:39:51.634424 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.633969 4866 scope.go:122] "RemoveContainer" containerID="10be2c01de270a3c93942abe29b60816d4edbe45088b247941c692cd1cf1c579" Nov 02 14:39:51.635364 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.635340 4866 generic.go:356] "Generic (PLEG): container finished" podID="9753ddaa-3899-46ea-952f-5f03d76037c6" containerID="e3f2beeb5ba2f2b69cee8f37a17604b46aba36ab4ba45c78a293f2fc46204c4e" exitCode=0 Nov 02 14:39:51.635459 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.635392 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" Nov 02 14:39:51.635459 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.635423 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" event={"ID":"9753ddaa-3899-46ea-952f-5f03d76037c6","Type":"ContainerDied","Data":"e3f2beeb5ba2f2b69cee8f37a17604b46aba36ab4ba45c78a293f2fc46204c4e"} Nov 02 14:39:51.635570 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.635471 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m" event={"ID":"9753ddaa-3899-46ea-952f-5f03d76037c6","Type":"ContainerDied","Data":"7ba3f58714731f68ee81bac359e0ffd71dc983650030f4313c1d3b754cc84d50"} Nov 02 14:39:51.637039 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.637022 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" event={"ID":"29dea1dd-e792-415a-9c82-dfa2edf71510","Type":"ContainerStarted","Data":"99a777fee782bccdb4f1cbf14410c1541ad8808b5bc85e4f67e0fd32d91494f9"} Nov 02 14:39:51.637132 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.637048 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" event={"ID":"29dea1dd-e792-415a-9c82-dfa2edf71510","Type":"ContainerStarted","Data":"6f410d52fde91b5522b3c8c77eea3315ad95ef838efbb5cdfc543d69f9c23f13"} Nov 02 14:39:51.644730 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.644559 4866 scope.go:122] "RemoveContainer" containerID="10be2c01de270a3c93942abe29b60816d4edbe45088b247941c692cd1cf1c579" Nov 02 14:39:51.644932 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:39:51.644902 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10be2c01de270a3c93942abe29b60816d4edbe45088b247941c692cd1cf1c579\": container with ID starting with 10be2c01de270a3c93942abe29b60816d4edbe45088b247941c692cd1cf1c579 not found: ID does not exist" containerID="10be2c01de270a3c93942abe29b60816d4edbe45088b247941c692cd1cf1c579" Nov 02 14:39:51.645040 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.644936 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10be2c01de270a3c93942abe29b60816d4edbe45088b247941c692cd1cf1c579"} err="failed to get container status \"10be2c01de270a3c93942abe29b60816d4edbe45088b247941c692cd1cf1c579\": rpc error: code = NotFound desc = could not find container \"10be2c01de270a3c93942abe29b60816d4edbe45088b247941c692cd1cf1c579\": container with ID starting with 10be2c01de270a3c93942abe29b60816d4edbe45088b247941c692cd1cf1c579 not found: ID does not exist" Nov 02 14:39:51.645040 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.644973 4866 scope.go:122] "RemoveContainer" containerID="e3f2beeb5ba2f2b69cee8f37a17604b46aba36ab4ba45c78a293f2fc46204c4e" Nov 02 14:39:51.654187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.654169 4866 scope.go:122] "RemoveContainer" containerID="e3f2beeb5ba2f2b69cee8f37a17604b46aba36ab4ba45c78a293f2fc46204c4e" Nov 02 14:39:51.654464 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:39:51.654439 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f2beeb5ba2f2b69cee8f37a17604b46aba36ab4ba45c78a293f2fc46204c4e\": container with ID starting with e3f2beeb5ba2f2b69cee8f37a17604b46aba36ab4ba45c78a293f2fc46204c4e not found: ID does not exist" containerID="e3f2beeb5ba2f2b69cee8f37a17604b46aba36ab4ba45c78a293f2fc46204c4e" Nov 02 14:39:51.654537 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.654474 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f2beeb5ba2f2b69cee8f37a17604b46aba36ab4ba45c78a293f2fc46204c4e"} err="failed to get container status \"e3f2beeb5ba2f2b69cee8f37a17604b46aba36ab4ba45c78a293f2fc46204c4e\": rpc error: code = NotFound desc = could not find container \"e3f2beeb5ba2f2b69cee8f37a17604b46aba36ab4ba45c78a293f2fc46204c4e\": container with ID starting with e3f2beeb5ba2f2b69cee8f37a17604b46aba36ab4ba45c78a293f2fc46204c4e not found: ID does not exist" Nov 02 14:39:51.685424 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.685390 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6"] Nov 02 14:39:51.697624 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.697584 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/distinct-w4-c0c28f-kserve-dc4bc6557-md4q6"] Nov 02 14:39:51.731270 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.731214 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" podStartSLOduration=2.731201129 podStartE2EDuration="2.731201129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:39:51.730027439 +0000 UTC m=+2556.913874777" watchObservedRunningTime="2026-11-02 14:39:51.731201129 +0000 UTC m=+2556.915048469" Nov 02 14:39:51.766994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.766958 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m"] Nov 02 14:39:51.772837 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:51.772804 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/simulator-w4-c0c28f-kserve-85dbf7fbb7-gtg7m"] Nov 02 14:39:52.340392 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:52.340356 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:39:52.945805 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:52.945771 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9753ddaa-3899-46ea-952f-5f03d76037c6" path="/var/lib/kubelet/pods/9753ddaa-3899-46ea-952f-5f03d76037c6/volumes" Nov 02 14:39:52.946247 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:52.946211 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7037eab-3f5f-419b-94f6-784ff7331925" path="/var/lib/kubelet/pods/d7037eab-3f5f-419b-94f6-784ff7331925/volumes" Nov 02 14:39:53.265887 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.265790 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" podUID="a76dbed2-c500-4d2e-8073-8b0936caecaf" containerName="main" containerID="cri-o://89dab39099ef576ac8f5aaad9200e25ba5873874ec4019152c034e9c4d722537" gracePeriod=45 Nov 02 14:39:53.270298 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.270265 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" podUID="a76dbed2-c500-4d2e-8073-8b0936caecaf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.139:8000/ready\": EOF" Nov 02 14:39:53.530820 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.530755 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:39:53.645095 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.645057 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-dshm\") pod \"a76dbed2-c500-4d2e-8073-8b0936caecaf\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " Nov 02 14:39:53.645304 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.645149 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a76dbed2-c500-4d2e-8073-8b0936caecaf-kube-api-access-jd2kd\" (UniqueName: \"kubernetes.io/projected/a76dbed2-c500-4d2e-8073-8b0936caecaf-kube-api-access-jd2kd\") pod \"a76dbed2-c500-4d2e-8073-8b0936caecaf\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " Nov 02 14:39:53.645304 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.645185 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-tmp-dir\") pod \"a76dbed2-c500-4d2e-8073-8b0936caecaf\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " Nov 02 14:39:53.645304 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.645285 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-model-cache\") pod \"a76dbed2-c500-4d2e-8073-8b0936caecaf\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " Nov 02 14:39:53.645470 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.645361 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-home\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-home\") pod \"a76dbed2-c500-4d2e-8073-8b0936caecaf\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " Nov 02 14:39:53.645470 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.645419 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a76dbed2-c500-4d2e-8073-8b0936caecaf-tls-certs\" (UniqueName: \"kubernetes.io/secret/a76dbed2-c500-4d2e-8073-8b0936caecaf-tls-certs\") pod \"a76dbed2-c500-4d2e-8073-8b0936caecaf\" (UID: \"a76dbed2-c500-4d2e-8073-8b0936caecaf\") " Nov 02 14:39:53.645574 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.645532 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-tmp-dir" pod "a76dbed2-c500-4d2e-8073-8b0936caecaf" (UID: "a76dbed2-c500-4d2e-8073-8b0936caecaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:53.645662 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.645638 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-home" pod "a76dbed2-c500-4d2e-8073-8b0936caecaf" (UID: "a76dbed2-c500-4d2e-8073-8b0936caecaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:53.645741 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.645656 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-model-cache" pod "a76dbed2-c500-4d2e-8073-8b0936caecaf" (UID: "a76dbed2-c500-4d2e-8073-8b0936caecaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:53.645983 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.645933 4866 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-tmp-dir\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:53.645983 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.645976 4866 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-model-cache\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:53.646165 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.645992 4866 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-home\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:53.647582 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.647541 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-dshm" pod "a76dbed2-c500-4d2e-8073-8b0936caecaf" (UID: "a76dbed2-c500-4d2e-8073-8b0936caecaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:53.647688 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.647670 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a76dbed2-c500-4d2e-8073-8b0936caecaf-tls-certs" pod "a76dbed2-c500-4d2e-8073-8b0936caecaf" (UID: "a76dbed2-c500-4d2e-8073-8b0936caecaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:39:53.647751 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.647689 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76dbed2-c500-4d2e-8073-8b0936caecaf-kube-api-access-jd2kd" pod "a76dbed2-c500-4d2e-8073-8b0936caecaf" (UID: "a76dbed2-c500-4d2e-8073-8b0936caecaf"). InnerVolumeSpecName "kube-api-access-jd2kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:39:53.651513 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.651487 4866 generic.go:356] "Generic (PLEG): container finished" podID="a76dbed2-c500-4d2e-8073-8b0936caecaf" containerID="89dab39099ef576ac8f5aaad9200e25ba5873874ec4019152c034e9c4d722537" exitCode=0 Nov 02 14:39:53.651604 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.651533 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" Nov 02 14:39:53.651604 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.651566 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" event={"ID":"a76dbed2-c500-4d2e-8073-8b0936caecaf","Type":"ContainerDied","Data":"89dab39099ef576ac8f5aaad9200e25ba5873874ec4019152c034e9c4d722537"} Nov 02 14:39:53.651692 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.651608 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth" event={"ID":"a76dbed2-c500-4d2e-8073-8b0936caecaf","Type":"ContainerDied","Data":"28187ca116e433f37e1047f600dac30d1adbc271f4e2aa38f811b90c8829a1c1"} Nov 02 14:39:53.651692 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.651626 4866 scope.go:122] "RemoveContainer" containerID="89dab39099ef576ac8f5aaad9200e25ba5873874ec4019152c034e9c4d722537" Nov 02 14:39:53.670363 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.670344 4866 scope.go:122] "RemoveContainer" containerID="89dab39099ef576ac8f5aaad9200e25ba5873874ec4019152c034e9c4d722537" Nov 02 14:39:53.670639 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:39:53.670618 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89dab39099ef576ac8f5aaad9200e25ba5873874ec4019152c034e9c4d722537\": container with ID starting with 89dab39099ef576ac8f5aaad9200e25ba5873874ec4019152c034e9c4d722537 not found: ID does not exist" containerID="89dab39099ef576ac8f5aaad9200e25ba5873874ec4019152c034e9c4d722537" Nov 02 14:39:53.670703 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.670647 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89dab39099ef576ac8f5aaad9200e25ba5873874ec4019152c034e9c4d722537"} err="failed to get container status \"89dab39099ef576ac8f5aaad9200e25ba5873874ec4019152c034e9c4d722537\": rpc error: code = NotFound desc = could not find container \"89dab39099ef576ac8f5aaad9200e25ba5873874ec4019152c034e9c4d722537\": container with ID starting with 89dab39099ef576ac8f5aaad9200e25ba5873874ec4019152c034e9c4d722537 not found: ID does not exist" Nov 02 14:39:53.712615 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.712580 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth"] Nov 02 14:39:53.718880 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.718849 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5ba53f/sub-test-model-5ba53f-kserve-56dc78469-d7kth"] Nov 02 14:39:53.747116 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.747079 4866 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76dbed2-c500-4d2e-8073-8b0936caecaf-tls-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:53.747116 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.747122 4866 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76dbed2-c500-4d2e-8073-8b0936caecaf-dshm\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:53.747381 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:53.747139 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd2kd\" (UniqueName: \"kubernetes.io/projected/a76dbed2-c500-4d2e-8073-8b0936caecaf-kube-api-access-jd2kd\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:39:54.945656 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:39:54.945619 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a76dbed2-c500-4d2e-8073-8b0936caecaf" path="/var/lib/kubelet/pods/a76dbed2-c500-4d2e-8073-8b0936caecaf/volumes" Nov 02 14:40:00.340415 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:00.340303 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:40:00.353564 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:00.353530 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:40:00.704551 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:00.704522 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:40:06.010358 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:40:06.010320 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:40:06.932604 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:06.932570 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw"] Nov 02 14:40:06.933129 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:06.933113 4866 cpu_manager.go:414] "containerMap: removing container" podUID="a76dbed2-c500-4d2e-8073-8b0936caecaf" containerName="main" Nov 02 14:40:06.933187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:06.933131 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a76dbed2-c500-4d2e-8073-8b0936caecaf" containerName="main" Nov 02 14:40:06.933187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:06.933158 4866 cpu_manager.go:414] "containerMap: removing container" podUID="9753ddaa-3899-46ea-952f-5f03d76037c6" containerName="main" Nov 02 14:40:06.933187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:06.933163 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9753ddaa-3899-46ea-952f-5f03d76037c6" containerName="main" Nov 02 14:40:06.933187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:06.933182 4866 cpu_manager.go:414] "containerMap: removing container" podUID="d7037eab-3f5f-419b-94f6-784ff7331925" containerName="main" Nov 02 14:40:06.933321 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:06.933189 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7037eab-3f5f-419b-94f6-784ff7331925" containerName="main" Nov 02 14:40:06.933321 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:06.933297 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="9753ddaa-3899-46ea-952f-5f03d76037c6" containerName="main" Nov 02 14:40:06.933321 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:06.933306 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7037eab-3f5f-419b-94f6-784ff7331925" containerName="main" Nov 02 14:40:06.933321 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:06.933317 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="a76dbed2-c500-4d2e-8073-8b0936caecaf" containerName="main" Nov 02 14:40:06.938382 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:06.938357 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:06.956999 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:06.956971 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw"] Nov 02 14:40:06.966233 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:06.966206 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-866159\"/\"e2e-conflict-b-be5aa686-kserve-self-signed-certs\"" Nov 02 14:40:06.983525 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:06.983493 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-model-cache\") pod \"e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:06.983668 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:06.983549 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-tmp-dir\") pod \"e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:06.983668 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:06.983646 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-dshm\") pod \"e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:06.983751 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:06.983669 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-tls-certs\") pod \"e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:06.983804 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:06.983786 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-home\") pod \"e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:06.983839 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:06.983814 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tz5z\" (UniqueName: \"kubernetes.io/projected/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-kube-api-access-5tz5z\") pod \"e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:07.084544 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:07.084507 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-dshm\") pod \"e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:07.084544 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:07.084547 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-tls-certs\") pod \"e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:07.085116 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:07.084630 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-home\") pod \"e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:07.085116 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:07.084661 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tz5z\" (UniqueName: \"kubernetes.io/projected/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-kube-api-access-5tz5z\") pod \"e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:07.085116 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:07.084707 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-model-cache\") pod \"e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:07.085116 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:07.084779 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-tmp-dir\") pod \"e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:07.085275 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:07.085118 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-home\") pod \"e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:07.085275 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:07.085134 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-tmp-dir\") pod \"e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:07.085275 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:07.085193 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-model-cache\") pod \"e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:07.086992 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:07.086969 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-dshm\") pod \"e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:07.087245 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:07.087224 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-tls-certs\") pod \"e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:07.109042 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:07.109013 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tz5z\" (UniqueName: \"kubernetes.io/projected/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-kube-api-access-5tz5z\") pod \"e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:07.251712 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:07.251678 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:07.550458 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:07.550433 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw"] Nov 02 14:40:07.552400 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:40:07.552376 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d5b2d9a_c9e4_4e9c_b080_7056e2016709.slice/crio-bf9c86c86bcafa062828658e425e4ee57d6248aa06a912c8c84d8b33fb4804d5 WatchSource:0}: Error finding container bf9c86c86bcafa062828658e425e4ee57d6248aa06a912c8c84d8b33fb4804d5: Status 404 returned error can't find the container with id bf9c86c86bcafa062828658e425e4ee57d6248aa06a912c8c84d8b33fb4804d5 Nov 02 14:40:07.734738 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:07.734704 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" event={"ID":"3d5b2d9a-c9e4-4e9c-b080-7056e2016709","Type":"ContainerStarted","Data":"542e5569004266522bd064cb2d198bc0e192b37d9bbe8040afaa5d41c4f1781f"} Nov 02 14:40:07.734738 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:07.734744 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" event={"ID":"3d5b2d9a-c9e4-4e9c-b080-7056e2016709","Type":"ContainerStarted","Data":"bf9c86c86bcafa062828658e425e4ee57d6248aa06a912c8c84d8b33fb4804d5"} Nov 02 14:40:07.780985 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:07.780918 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" podStartSLOduration=1.780904171 podStartE2EDuration="1.780904171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:40:07.779863716 +0000 UTC m=+2572.963711055" watchObservedRunningTime="2026-11-02 14:40:07.780904171 +0000 UTC m=+2572.964751507" Nov 02 14:40:08.252603 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:08.252565 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:11.538978 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:11.538930 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf"] Nov 02 14:40:11.539475 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:11.539192 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" podUID="a3dcadd1-584c-4dee-b0c0-402f8b4175c5" containerName="maas-api" containerID="cri-o://d933858ddebde1fe38a03b1affd300a87ab601c04a154138eb6be74f17650765" gracePeriod=30 Nov 02 14:40:11.758263 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:11.758116 4866 generic.go:356] "Generic (PLEG): container finished" podID="a3dcadd1-584c-4dee-b0c0-402f8b4175c5" containerID="d933858ddebde1fe38a03b1affd300a87ab601c04a154138eb6be74f17650765" exitCode=0 Nov 02 14:40:11.758263 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:11.758164 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" event={"ID":"a3dcadd1-584c-4dee-b0c0-402f8b4175c5","Type":"ContainerDied","Data":"d933858ddebde1fe38a03b1affd300a87ab601c04a154138eb6be74f17650765"} Nov 02 14:40:11.816341 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:11.816309 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" Nov 02 14:40:11.938580 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:11.938545 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-metrics-certs\") pod \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\" (UID: \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\") " Nov 02 14:40:11.938796 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:11.938604 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-openshift-service-ca\") pod \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\" (UID: \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\") " Nov 02 14:40:11.938796 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:11.938661 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-maas-api-tls\") pod \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\" (UID: \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\") " Nov 02 14:40:11.938796 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:11.938757 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-kube-api-access-2q7d7\" (UniqueName: \"kubernetes.io/projected/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-kube-api-access-2q7d7\") pod \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\" (UID: \"a3dcadd1-584c-4dee-b0c0-402f8b4175c5\") " Nov 02 14:40:11.939011 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:11.938987 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-openshift-service-ca" pod "a3dcadd1-584c-4dee-b0c0-402f8b4175c5" (UID: "a3dcadd1-584c-4dee-b0c0-402f8b4175c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:40:11.939266 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:11.939240 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:11.940806 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:11.940780 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-maas-api-tls" pod "a3dcadd1-584c-4dee-b0c0-402f8b4175c5" (UID: "a3dcadd1-584c-4dee-b0c0-402f8b4175c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:40:11.940919 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:11.940807 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-kube-api-access-2q7d7" pod "a3dcadd1-584c-4dee-b0c0-402f8b4175c5" (UID: "a3dcadd1-584c-4dee-b0c0-402f8b4175c5"). InnerVolumeSpecName "kube-api-access-2q7d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:40:11.940919 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:11.940858 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-metrics-certs" pod "a3dcadd1-584c-4dee-b0c0-402f8b4175c5" (UID: "a3dcadd1-584c-4dee-b0c0-402f8b4175c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:40:12.040962 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:12.040831 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:12.040962 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:12.040876 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:12.040962 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:12.040891 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q7d7\" (UniqueName: \"kubernetes.io/projected/a3dcadd1-584c-4dee-b0c0-402f8b4175c5-kube-api-access-2q7d7\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:12.764613 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:12.764534 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" Nov 02 14:40:12.764613 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:12.764568 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf" event={"ID":"a3dcadd1-584c-4dee-b0c0-402f8b4175c5","Type":"ContainerDied","Data":"68043d61c1bb7cfba261b2ac95374171fdb2627b9f5040a3fedde1b0ff16ee35"} Nov 02 14:40:12.765108 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:12.764633 4866 scope.go:122] "RemoveContainer" containerID="d933858ddebde1fe38a03b1affd300a87ab601c04a154138eb6be74f17650765" Nov 02 14:40:12.816794 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:12.816760 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf"] Nov 02 14:40:12.828440 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:12.828402 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c0c28f-679c75986-nh4zf"] Nov 02 14:40:12.946365 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:12.946333 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3dcadd1-584c-4dee-b0c0-402f8b4175c5" path="/var/lib/kubelet/pods/a3dcadd1-584c-4dee-b0c0-402f8b4175c5/volumes" Nov 02 14:40:16.178398 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:16.178363 4866 scope.go:122] "RemoveContainer" containerID="af1ae0442c88c59475cb4dcd62b7aa56b25dcd590d3c803c17cee03085b3ad8f" Nov 02 14:40:16.187891 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:16.187868 4866 scope.go:122] "RemoveContainer" containerID="13503b7ba366e7a57736d4792dff24adb26c33072a47f4bf4c5f0a978ae8c6f2" Nov 02 14:40:17.252217 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:17.252178 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:17.264627 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:17.264597 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:17.804281 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:17.804251 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:30.835131 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:30.835091 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr"] Nov 02 14:40:30.835664 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:30.835644 4866 cpu_manager.go:414] "containerMap: removing container" podUID="a3dcadd1-584c-4dee-b0c0-402f8b4175c5" containerName="maas-api" Nov 02 14:40:30.835708 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:30.835667 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3dcadd1-584c-4dee-b0c0-402f8b4175c5" containerName="maas-api" Nov 02 14:40:30.835804 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:30.835791 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3dcadd1-584c-4dee-b0c0-402f8b4175c5" containerName="maas-api" Nov 02 14:40:30.841301 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:30.841278 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:30.862578 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:30.862547 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-866159-openshift-default-dockercfg-jpxc8\"" Nov 02 14:40:30.877588 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:30.877560 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr"] Nov 02 14:40:30.960872 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:30.960829 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:30.960872 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:30.960887 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:30.961190 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:30.960912 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:30.961190 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:30.961091 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:30.961190 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:30.961145 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:30.961294 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:30.961231 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fpk6\" (UniqueName: \"kubernetes.io/projected/d94980c9-4b40-448d-9e6d-1a8176035b8e-kube-api-access-5fpk6\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:30.961294 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:30.961264 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94980c9-4b40-448d-9e6d-1a8176035b8e-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:30.961294 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:30.961292 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:30.961440 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:30.961406 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.062434 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.062399 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.062633 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.062472 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.062633 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.062517 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fpk6\" (UniqueName: \"kubernetes.io/projected/d94980c9-4b40-448d-9e6d-1a8176035b8e-kube-api-access-5fpk6\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.062633 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.062560 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94980c9-4b40-448d-9e6d-1a8176035b8e-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.062633 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.062609 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.062883 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.062673 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.062883 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.062757 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.062883 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.062844 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.062883 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.062865 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.063108 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.062892 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.063108 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.062906 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.063237 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.063191 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.063435 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.063411 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.063530 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.063461 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94980c9-4b40-448d-9e6d-1a8176035b8e-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.065062 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.065035 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.065423 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.065405 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.085760 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.085700 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.085911 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.085892 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fpk6\" (UniqueName: \"kubernetes.io/projected/d94980c9-4b40-448d-9e6d-1a8176035b8e-kube-api-access-5fpk6\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.155157 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.155118 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:31.480034 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.480007 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr"] Nov 02 14:40:31.481652 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:40:31.481623 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94980c9_4b40_448d_9e6d_1a8176035b8e.slice/crio-f234e01a824cfb21532b99eab2fabac097282d01ecdf5c45454930a9ea99546a WatchSource:0}: Error finding container f234e01a824cfb21532b99eab2fabac097282d01ecdf5c45454930a9ea99546a: Status 404 returned error can't find the container with id f234e01a824cfb21532b99eab2fabac097282d01ecdf5c45454930a9ea99546a Nov 02 14:40:31.484540 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.484507 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:40:31.484629 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.484579 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:40:31.484629 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.484608 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:40:31.885190 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.885093 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" event={"ID":"d94980c9-4b40-448d-9e6d-1a8176035b8e","Type":"ContainerStarted","Data":"80d396d4def9423894c9004e8368850ac3c97cfce9ed341945f07513243dd985"} Nov 02 14:40:31.885190 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.885153 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" event={"ID":"d94980c9-4b40-448d-9e6d-1a8176035b8e","Type":"ContainerStarted","Data":"f234e01a824cfb21532b99eab2fabac097282d01ecdf5c45454930a9ea99546a"} Nov 02 14:40:31.932898 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:31.932820 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" podStartSLOduration=1.93279828 podStartE2EDuration="1.93279828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:40:31.931770157 +0000 UTC m=+2597.115617497" watchObservedRunningTime="2026-11-02 14:40:31.93279828 +0000 UTC m=+2597.116645619" Nov 02 14:40:32.155413 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:32.155323 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:32.156998 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:32.156969 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 14:40:32.157130 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:32.157022 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" podUID="d94980c9-4b40-448d-9e6d-1a8176035b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 14:40:32.836483 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:32.836428 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw"] Nov 02 14:40:33.156074 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:33.155939 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 14:40:33.156074 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:33.156035 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" podUID="d94980c9-4b40-448d-9e6d-1a8176035b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 14:40:34.156043 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:34.156004 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 14:40:34.156229 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:34.156059 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" podUID="d94980c9-4b40-448d-9e6d-1a8176035b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 14:40:34.563126 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:34.563085 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh"] Nov 02 14:40:35.156269 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:35.156232 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 14:40:35.156657 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:35.156289 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" podUID="d94980c9-4b40-448d-9e6d-1a8176035b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 14:40:36.031401 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:40:36.031364 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:40:36.156227 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:36.156184 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 14:40:36.156411 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:36.156248 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" podUID="d94980c9-4b40-448d-9e6d-1a8176035b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 14:40:37.156161 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:37.156121 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 14:40:37.156379 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:37.156188 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" podUID="d94980c9-4b40-448d-9e6d-1a8176035b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 14:40:38.156326 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:38.156286 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 02 14:40:38.156734 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:38.156344 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" podUID="d94980c9-4b40-448d-9e6d-1a8176035b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 02 14:40:39.281696 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:39.281663 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:39.282157 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:39.281973 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:39.283082 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:39.283056 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:40:41.655615 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:41.655571 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2"] Nov 02 14:40:41.656195 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:41.655831 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" podUID="6803eccc-186b-4560-9dab-82bee489a940" containerName="maas-api" containerID="cri-o://12848d78d5ca350f59f6e2a798d60a0e89d5b78ab9c999286d62b7441568e068" gracePeriod=30 Nov 02 14:40:41.763883 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:41.763844 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5"] Nov 02 14:40:41.938810 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:41.938785 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" Nov 02 14:40:41.967297 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:41.967269 4866 generic.go:356] "Generic (PLEG): container finished" podID="6803eccc-186b-4560-9dab-82bee489a940" containerID="12848d78d5ca350f59f6e2a798d60a0e89d5b78ab9c999286d62b7441568e068" exitCode=0 Nov 02 14:40:41.967475 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:41.967322 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" Nov 02 14:40:41.967475 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:41.967345 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" event={"ID":"6803eccc-186b-4560-9dab-82bee489a940","Type":"ContainerDied","Data":"12848d78d5ca350f59f6e2a798d60a0e89d5b78ab9c999286d62b7441568e068"} Nov 02 14:40:41.967475 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:41.967396 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2" event={"ID":"6803eccc-186b-4560-9dab-82bee489a940","Type":"ContainerDied","Data":"fcb54ccd7c1cfd102bc13b50433f126adad7a16a07f884579349a02e7923ab7d"} Nov 02 14:40:41.967475 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:41.967418 4866 scope.go:122] "RemoveContainer" containerID="12848d78d5ca350f59f6e2a798d60a0e89d5b78ab9c999286d62b7441568e068" Nov 02 14:40:41.976627 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:41.976607 4866 scope.go:122] "RemoveContainer" containerID="12848d78d5ca350f59f6e2a798d60a0e89d5b78ab9c999286d62b7441568e068" Nov 02 14:40:41.976916 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:40:41.976892 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12848d78d5ca350f59f6e2a798d60a0e89d5b78ab9c999286d62b7441568e068\": container with ID starting with 12848d78d5ca350f59f6e2a798d60a0e89d5b78ab9c999286d62b7441568e068 not found: ID does not exist" containerID="12848d78d5ca350f59f6e2a798d60a0e89d5b78ab9c999286d62b7441568e068" Nov 02 14:40:41.976987 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:41.976924 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12848d78d5ca350f59f6e2a798d60a0e89d5b78ab9c999286d62b7441568e068"} err="failed to get container status \"12848d78d5ca350f59f6e2a798d60a0e89d5b78ab9c999286d62b7441568e068\": rpc error: code = NotFound desc = could not find container \"12848d78d5ca350f59f6e2a798d60a0e89d5b78ab9c999286d62b7441568e068\": container with ID starting with 12848d78d5ca350f59f6e2a798d60a0e89d5b78ab9c999286d62b7441568e068 not found: ID does not exist" Nov 02 14:40:42.019670 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:42.019629 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6803eccc-186b-4560-9dab-82bee489a940-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6803eccc-186b-4560-9dab-82bee489a940-metrics-certs\") pod \"6803eccc-186b-4560-9dab-82bee489a940\" (UID: \"6803eccc-186b-4560-9dab-82bee489a940\") " Nov 02 14:40:42.019887 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:42.019743 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6803eccc-186b-4560-9dab-82bee489a940-kube-api-access-t6lzl\" (UniqueName: \"kubernetes.io/projected/6803eccc-186b-4560-9dab-82bee489a940-kube-api-access-t6lzl\") pod \"6803eccc-186b-4560-9dab-82bee489a940\" (UID: \"6803eccc-186b-4560-9dab-82bee489a940\") " Nov 02 14:40:42.019887 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:42.019814 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6803eccc-186b-4560-9dab-82bee489a940-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6803eccc-186b-4560-9dab-82bee489a940-maas-api-tls\") pod \"6803eccc-186b-4560-9dab-82bee489a940\" (UID: \"6803eccc-186b-4560-9dab-82bee489a940\") " Nov 02 14:40:42.019887 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:42.019872 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6803eccc-186b-4560-9dab-82bee489a940-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6803eccc-186b-4560-9dab-82bee489a940-openshift-service-ca\") pod \"6803eccc-186b-4560-9dab-82bee489a940\" (UID: \"6803eccc-186b-4560-9dab-82bee489a940\") " Nov 02 14:40:42.020342 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:42.020309 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6803eccc-186b-4560-9dab-82bee489a940-openshift-service-ca" pod "6803eccc-186b-4560-9dab-82bee489a940" (UID: "6803eccc-186b-4560-9dab-82bee489a940"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:40:42.021937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:42.021908 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6803eccc-186b-4560-9dab-82bee489a940-maas-api-tls" pod "6803eccc-186b-4560-9dab-82bee489a940" (UID: "6803eccc-186b-4560-9dab-82bee489a940"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:40:42.021937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:42.021922 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6803eccc-186b-4560-9dab-82bee489a940-metrics-certs" pod "6803eccc-186b-4560-9dab-82bee489a940" (UID: "6803eccc-186b-4560-9dab-82bee489a940"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:40:42.022272 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:42.022254 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6803eccc-186b-4560-9dab-82bee489a940-kube-api-access-t6lzl" pod "6803eccc-186b-4560-9dab-82bee489a940" (UID: "6803eccc-186b-4560-9dab-82bee489a940"). InnerVolumeSpecName "kube-api-access-t6lzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:40:42.121533 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:42.121491 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6803eccc-186b-4560-9dab-82bee489a940-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:42.121533 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:42.121527 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6lzl\" (UniqueName: \"kubernetes.io/projected/6803eccc-186b-4560-9dab-82bee489a940-kube-api-access-t6lzl\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:42.121533 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:42.121538 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6803eccc-186b-4560-9dab-82bee489a940-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:42.121775 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:42.121547 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6803eccc-186b-4560-9dab-82bee489a940-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:42.314178 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:42.314144 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2"] Nov 02 14:40:42.316338 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:42.316308 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e07583-69d9cbb67b-qt5c2"] Nov 02 14:40:42.945686 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:42.945652 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6803eccc-186b-4560-9dab-82bee489a940" path="/var/lib/kubelet/pods/6803eccc-186b-4560-9dab-82bee489a940/volumes" Nov 02 14:40:46.764231 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:46.764161 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" podUID="2d243c72-dc2f-458e-8f7b-23f89144e7bf" containerName="payload-pre-processing" containerID="cri-o://4621448a13b0db718dd710e88f30e200a2c16a31fd09008b226711c9b666acab" gracePeriod=25 Nov 02 14:40:47.858874 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:47.858829 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" podUID="3d5b2d9a-c9e4-4e9c-b080-7056e2016709" containerName="main" containerID="cri-o://542e5569004266522bd064cb2d198bc0e192b37d9bbe8040afaa5d41c4f1781f" gracePeriod=45 Nov 02 14:40:48.002233 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.002196 4866 generic.go:356] "Generic (PLEG): container finished" podID="3d5b2d9a-c9e4-4e9c-b080-7056e2016709" containerID="542e5569004266522bd064cb2d198bc0e192b37d9bbe8040afaa5d41c4f1781f" exitCode=0 Nov 02 14:40:48.002415 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.002265 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" event={"ID":"3d5b2d9a-c9e4-4e9c-b080-7056e2016709","Type":"ContainerDied","Data":"542e5569004266522bd064cb2d198bc0e192b37d9bbe8040afaa5d41c4f1781f"} Nov 02 14:40:48.133180 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.133153 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:48.184616 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.184573 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-kube-api-access-5tz5z\" (UniqueName: \"kubernetes.io/projected/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-kube-api-access-5tz5z\") pod \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " Nov 02 14:40:48.184807 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.184673 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-dshm\") pod \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " Nov 02 14:40:48.184807 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.184754 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-home\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-home\") pod \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " Nov 02 14:40:48.184924 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.184802 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-model-cache\") pod \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " Nov 02 14:40:48.184924 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.184856 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-tmp-dir\") pod \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " Nov 02 14:40:48.184924 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.184894 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-tls-certs\") pod \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\" (UID: \"3d5b2d9a-c9e4-4e9c-b080-7056e2016709\") " Nov 02 14:40:48.186075 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.186043 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-home" pod "3d5b2d9a-c9e4-4e9c-b080-7056e2016709" (UID: "3d5b2d9a-c9e4-4e9c-b080-7056e2016709"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:48.186477 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.186429 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-model-cache" pod "3d5b2d9a-c9e4-4e9c-b080-7056e2016709" (UID: "3d5b2d9a-c9e4-4e9c-b080-7056e2016709"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:48.186621 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.186465 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-tmp-dir" pod "3d5b2d9a-c9e4-4e9c-b080-7056e2016709" (UID: "3d5b2d9a-c9e4-4e9c-b080-7056e2016709"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:48.187984 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.187913 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-kube-api-access-5tz5z" pod "3d5b2d9a-c9e4-4e9c-b080-7056e2016709" (UID: "3d5b2d9a-c9e4-4e9c-b080-7056e2016709"). InnerVolumeSpecName "kube-api-access-5tz5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:40:48.188262 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.188237 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-tls-certs" pod "3d5b2d9a-c9e4-4e9c-b080-7056e2016709" (UID: "3d5b2d9a-c9e4-4e9c-b080-7056e2016709"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:40:48.188677 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.188652 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-dshm" pod "3d5b2d9a-c9e4-4e9c-b080-7056e2016709" (UID: "3d5b2d9a-c9e4-4e9c-b080-7056e2016709"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:48.287155 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.287112 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tz5z\" (UniqueName: \"kubernetes.io/projected/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-kube-api-access-5tz5z\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:48.287155 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.287150 4866 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-dshm\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:48.287155 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.287161 4866 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-home\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:48.287401 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.287170 4866 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-model-cache\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:48.287401 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.287202 4866 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-tmp-dir\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:48.287401 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:48.287211 4866 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5b2d9a-c9e4-4e9c-b080-7056e2016709-tls-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:49.009963 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.009921 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" Nov 02 14:40:49.010481 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.009925 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw" event={"ID":"3d5b2d9a-c9e4-4e9c-b080-7056e2016709","Type":"ContainerDied","Data":"bf9c86c86bcafa062828658e425e4ee57d6248aa06a912c8c84d8b33fb4804d5"} Nov 02 14:40:49.010481 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.010054 4866 scope.go:122] "RemoveContainer" containerID="542e5569004266522bd064cb2d198bc0e192b37d9bbe8040afaa5d41c4f1781f" Nov 02 14:40:49.060653 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.060620 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw"] Nov 02 14:40:49.072289 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.072239 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-866159/e2e-conflict-b-be5aa686-kserve-6d4bbd6d8c-tmgqw"] Nov 02 14:40:49.583386 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.583348 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" podUID="29dea1dd-e792-415a-9c82-dfa2edf71510" containerName="main" containerID="cri-o://99a777fee782bccdb4f1cbf14410c1541ad8808b5bc85e4f67e0fd32d91494f9" gracePeriod=45 Nov 02 14:40:49.694626 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.694582 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" podUID="29dea1dd-e792-415a-9c82-dfa2edf71510" containerName="main" probeResult="failure" output="Get \"https://10.132.0.144:8000/ready\": dial tcp 10.132.0.144:8000: connect: connection refused" Nov 02 14:40:49.846167 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.846106 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:40:49.854114 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.854077 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" podUID="2d243c72-dc2f-458e-8f7b-23f89144e7bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:40:49.902251 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.902203 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29dea1dd-e792-415a-9c82-dfa2edf71510-tls-certs\" (UniqueName: \"kubernetes.io/secret/29dea1dd-e792-415a-9c82-dfa2edf71510-tls-certs\") pod \"29dea1dd-e792-415a-9c82-dfa2edf71510\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " Nov 02 14:40:49.902512 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.902486 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-tmp-dir\") pod \"29dea1dd-e792-415a-9c82-dfa2edf71510\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " Nov 02 14:40:49.902658 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.902579 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-model-cache\") pod \"29dea1dd-e792-415a-9c82-dfa2edf71510\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " Nov 02 14:40:49.902723 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.902663 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-dshm\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-dshm\") pod \"29dea1dd-e792-415a-9c82-dfa2edf71510\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " Nov 02 14:40:49.902723 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.902690 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-home\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-home\") pod \"29dea1dd-e792-415a-9c82-dfa2edf71510\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " Nov 02 14:40:49.902837 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.902733 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29dea1dd-e792-415a-9c82-dfa2edf71510-kube-api-access-msrb7\" (UniqueName: \"kubernetes.io/projected/29dea1dd-e792-415a-9c82-dfa2edf71510-kube-api-access-msrb7\") pod \"29dea1dd-e792-415a-9c82-dfa2edf71510\" (UID: \"29dea1dd-e792-415a-9c82-dfa2edf71510\") " Nov 02 14:40:49.902892 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.902847 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-tmp-dir" pod "29dea1dd-e792-415a-9c82-dfa2edf71510" (UID: "29dea1dd-e792-415a-9c82-dfa2edf71510"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:49.903333 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.903168 4866 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-tmp-dir\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:49.904085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.904060 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-home" pod "29dea1dd-e792-415a-9c82-dfa2edf71510" (UID: "29dea1dd-e792-415a-9c82-dfa2edf71510"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:49.905183 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.905140 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-model-cache" pod "29dea1dd-e792-415a-9c82-dfa2edf71510" (UID: "29dea1dd-e792-415a-9c82-dfa2edf71510"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:49.908524 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.908492 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-dshm" pod "29dea1dd-e792-415a-9c82-dfa2edf71510" (UID: "29dea1dd-e792-415a-9c82-dfa2edf71510"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:49.908664 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.908519 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29dea1dd-e792-415a-9c82-dfa2edf71510-tls-certs" pod "29dea1dd-e792-415a-9c82-dfa2edf71510" (UID: "29dea1dd-e792-415a-9c82-dfa2edf71510"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:40:49.908836 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:49.908808 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29dea1dd-e792-415a-9c82-dfa2edf71510-kube-api-access-msrb7" pod "29dea1dd-e792-415a-9c82-dfa2edf71510" (UID: "29dea1dd-e792-415a-9c82-dfa2edf71510"). InnerVolumeSpecName "kube-api-access-msrb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:40:50.004164 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:50.004118 4866 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-model-cache\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:50.004164 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:50.004157 4866 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-dshm\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:50.004164 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:50.004169 4866 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29dea1dd-e792-415a-9c82-dfa2edf71510-home\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:50.004412 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:50.004184 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msrb7\" (UniqueName: \"kubernetes.io/projected/29dea1dd-e792-415a-9c82-dfa2edf71510-kube-api-access-msrb7\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:50.004412 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:50.004197 4866 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29dea1dd-e792-415a-9c82-dfa2edf71510-tls-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:40:50.016703 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:50.016670 4866 generic.go:356] "Generic (PLEG): container finished" podID="29dea1dd-e792-415a-9c82-dfa2edf71510" containerID="99a777fee782bccdb4f1cbf14410c1541ad8808b5bc85e4f67e0fd32d91494f9" exitCode=0 Nov 02 14:40:50.017121 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:50.016718 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" Nov 02 14:40:50.017121 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:50.016752 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" event={"ID":"29dea1dd-e792-415a-9c82-dfa2edf71510","Type":"ContainerDied","Data":"99a777fee782bccdb4f1cbf14410c1541ad8808b5bc85e4f67e0fd32d91494f9"} Nov 02 14:40:50.017121 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:50.016812 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh" event={"ID":"29dea1dd-e792-415a-9c82-dfa2edf71510","Type":"ContainerDied","Data":"6f410d52fde91b5522b3c8c77eea3315ad95ef838efbb5cdfc543d69f9c23f13"} Nov 02 14:40:50.017121 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:50.016837 4866 scope.go:122] "RemoveContainer" containerID="99a777fee782bccdb4f1cbf14410c1541ad8808b5bc85e4f67e0fd32d91494f9" Nov 02 14:40:50.026713 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:50.026685 4866 scope.go:122] "RemoveContainer" containerID="99a777fee782bccdb4f1cbf14410c1541ad8808b5bc85e4f67e0fd32d91494f9" Nov 02 14:40:50.027018 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:40:50.026999 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a777fee782bccdb4f1cbf14410c1541ad8808b5bc85e4f67e0fd32d91494f9\": container with ID starting with 99a777fee782bccdb4f1cbf14410c1541ad8808b5bc85e4f67e0fd32d91494f9 not found: ID does not exist" containerID="99a777fee782bccdb4f1cbf14410c1541ad8808b5bc85e4f67e0fd32d91494f9" Nov 02 14:40:50.027085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:50.027027 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a777fee782bccdb4f1cbf14410c1541ad8808b5bc85e4f67e0fd32d91494f9"} err="failed to get container status \"99a777fee782bccdb4f1cbf14410c1541ad8808b5bc85e4f67e0fd32d91494f9\": rpc error: code = NotFound desc = could not find container \"99a777fee782bccdb4f1cbf14410c1541ad8808b5bc85e4f67e0fd32d91494f9\": container with ID starting with 99a777fee782bccdb4f1cbf14410c1541ad8808b5bc85e4f67e0fd32d91494f9 not found: ID does not exist" Nov 02 14:40:50.069423 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:50.069386 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh"] Nov 02 14:40:50.071848 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:50.071819 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-866159/e2e-conflict-a-be5aa686-kserve-654f546949-p9rwh"] Nov 02 14:40:50.945548 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:50.945514 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29dea1dd-e792-415a-9c82-dfa2edf71510" path="/var/lib/kubelet/pods/29dea1dd-e792-415a-9c82-dfa2edf71510/volumes" Nov 02 14:40:50.945967 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:50.945936 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d5b2d9a-c9e4-4e9c-b080-7056e2016709" path="/var/lib/kubelet/pods/3d5b2d9a-c9e4-4e9c-b080-7056e2016709/volumes" Nov 02 14:40:59.853727 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:40:59.853683 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" podUID="2d243c72-dc2f-458e-8f7b-23f89144e7bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:41:00.156237 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:00.156147 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z"] Nov 02 14:41:05.157269 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:05.157193 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" podUID="dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8" containerName="payload-pre-processing" containerID="cri-o://4fe080d0bbcd7faf854b30b02e62087a5ac40cd696b4e2c49646f10a08cdcdf3" gracePeriod=25 Nov 02 14:41:06.011387 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:41:06.011341 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:41:07.023723 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:07.023663 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" Nov 02 14:41:07.138313 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:07.138276 4866 generic.go:356] "Generic (PLEG): container finished" podID="2d243c72-dc2f-458e-8f7b-23f89144e7bf" containerID="4621448a13b0db718dd710e88f30e200a2c16a31fd09008b226711c9b666acab" exitCode=0 Nov 02 14:41:07.138555 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:07.138353 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" event={"ID":"2d243c72-dc2f-458e-8f7b-23f89144e7bf","Type":"ContainerDied","Data":"4621448a13b0db718dd710e88f30e200a2c16a31fd09008b226711c9b666acab"} Nov 02 14:41:07.138555 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:07.138404 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" event={"ID":"2d243c72-dc2f-458e-8f7b-23f89144e7bf","Type":"ContainerDied","Data":"59288b47122cc17e549799eaf7c1dd6a899645997b8617e8a4249043ef53a199"} Nov 02 14:41:07.138555 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:07.138427 4866 scope.go:122] "RemoveContainer" containerID="4621448a13b0db718dd710e88f30e200a2c16a31fd09008b226711c9b666acab" Nov 02 14:41:07.138742 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:07.138723 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5" Nov 02 14:41:07.149730 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:07.149706 4866 scope.go:122] "RemoveContainer" containerID="4621448a13b0db718dd710e88f30e200a2c16a31fd09008b226711c9b666acab" Nov 02 14:41:07.150098 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:41:07.150066 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4621448a13b0db718dd710e88f30e200a2c16a31fd09008b226711c9b666acab\": container with ID starting with 4621448a13b0db718dd710e88f30e200a2c16a31fd09008b226711c9b666acab not found: ID does not exist" containerID="4621448a13b0db718dd710e88f30e200a2c16a31fd09008b226711c9b666acab" Nov 02 14:41:07.150221 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:07.150104 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4621448a13b0db718dd710e88f30e200a2c16a31fd09008b226711c9b666acab"} err="failed to get container status \"4621448a13b0db718dd710e88f30e200a2c16a31fd09008b226711c9b666acab\": rpc error: code = NotFound desc = could not find container \"4621448a13b0db718dd710e88f30e200a2c16a31fd09008b226711c9b666acab\": container with ID starting with 4621448a13b0db718dd710e88f30e200a2c16a31fd09008b226711c9b666acab not found: ID does not exist" Nov 02 14:41:07.190086 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:07.190052 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d243c72-dc2f-458e-8f7b-23f89144e7bf-kube-api-access-rzdqk\" (UniqueName: \"kubernetes.io/projected/2d243c72-dc2f-458e-8f7b-23f89144e7bf-kube-api-access-rzdqk\") pod \"2d243c72-dc2f-458e-8f7b-23f89144e7bf\" (UID: \"2d243c72-dc2f-458e-8f7b-23f89144e7bf\") " Nov 02 14:41:07.190290 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:07.190191 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d243c72-dc2f-458e-8f7b-23f89144e7bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d243c72-dc2f-458e-8f7b-23f89144e7bf-plugins-config-volume\") pod \"2d243c72-dc2f-458e-8f7b-23f89144e7bf\" (UID: \"2d243c72-dc2f-458e-8f7b-23f89144e7bf\") " Nov 02 14:41:07.190664 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:07.190634 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d243c72-dc2f-458e-8f7b-23f89144e7bf-plugins-config-volume" pod "2d243c72-dc2f-458e-8f7b-23f89144e7bf" (UID: "2d243c72-dc2f-458e-8f7b-23f89144e7bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:41:07.192501 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:07.192470 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d243c72-dc2f-458e-8f7b-23f89144e7bf-kube-api-access-rzdqk" pod "2d243c72-dc2f-458e-8f7b-23f89144e7bf" (UID: "2d243c72-dc2f-458e-8f7b-23f89144e7bf"). InnerVolumeSpecName "kube-api-access-rzdqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:41:07.291872 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:07.291778 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d243c72-dc2f-458e-8f7b-23f89144e7bf-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:07.291872 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:07.291815 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzdqk\" (UniqueName: \"kubernetes.io/projected/2d243c72-dc2f-458e-8f7b-23f89144e7bf-kube-api-access-rzdqk\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:07.485296 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:07.485260 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5"] Nov 02 14:41:07.495540 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:07.495503 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e07583-65b978697f-hjgn5"] Nov 02 14:41:08.945660 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:08.945626 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d243c72-dc2f-458e-8f7b-23f89144e7bf" path="/var/lib/kubelet/pods/2d243c72-dc2f-458e-8f7b-23f89144e7bf/volumes" Nov 02 14:41:11.377104 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:11.377062 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" podUID="dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:41:21.376752 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:21.376712 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" podUID="dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:41:25.114871 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:25.114824 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l"] Nov 02 14:41:25.115420 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:25.115143 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" podUID="6bc409ca-3338-4474-bad2-aeb03d23aedc" containerName="istio-proxy" containerID="cri-o://f305c6bde605dddd1576a2ebfd1a7acb02a443b2c4eeaf6f1ca40ed435648e6c" gracePeriod=30 Nov 02 14:41:25.412412 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:25.412387 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" Nov 02 14:41:25.469601 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:25.469554 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8-plugins-config-volume\") pod \"dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8\" (UID: \"dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8\") " Nov 02 14:41:25.469791 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:25.469610 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8-kube-api-access-j9rpz\" (UniqueName: \"kubernetes.io/projected/dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8-kube-api-access-j9rpz\") pod \"dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8\" (UID: \"dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8\") " Nov 02 14:41:25.470004 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:25.469980 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8-plugins-config-volume" pod "dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8" (UID: "dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:41:25.472187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:25.472156 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8-kube-api-access-j9rpz" pod "dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8" (UID: "dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8"). InnerVolumeSpecName "kube-api-access-j9rpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:41:25.571177 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:25.571138 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:25.571177 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:25.571168 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9rpz\" (UniqueName: \"kubernetes.io/projected/dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8-kube-api-access-j9rpz\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:26.255617 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:26.255580 4866 generic.go:356] "Generic (PLEG): container finished" podID="dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8" containerID="4fe080d0bbcd7faf854b30b02e62087a5ac40cd696b4e2c49646f10a08cdcdf3" exitCode=0 Nov 02 14:41:26.256068 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:26.255631 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" Nov 02 14:41:26.256068 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:26.255660 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" event={"ID":"dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8","Type":"ContainerDied","Data":"4fe080d0bbcd7faf854b30b02e62087a5ac40cd696b4e2c49646f10a08cdcdf3"} Nov 02 14:41:26.256068 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:26.255722 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z" event={"ID":"dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8","Type":"ContainerDied","Data":"1c45a53c0b5086694f570c8b8121c14e7c7983ac2f9e6ecc72c72835cc2cceff"} Nov 02 14:41:26.256068 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:26.255744 4866 scope.go:122] "RemoveContainer" containerID="4fe080d0bbcd7faf854b30b02e62087a5ac40cd696b4e2c49646f10a08cdcdf3" Nov 02 14:41:26.265716 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:26.265695 4866 scope.go:122] "RemoveContainer" containerID="4fe080d0bbcd7faf854b30b02e62087a5ac40cd696b4e2c49646f10a08cdcdf3" Nov 02 14:41:26.266101 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:41:26.266078 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fe080d0bbcd7faf854b30b02e62087a5ac40cd696b4e2c49646f10a08cdcdf3\": container with ID starting with 4fe080d0bbcd7faf854b30b02e62087a5ac40cd696b4e2c49646f10a08cdcdf3 not found: ID does not exist" containerID="4fe080d0bbcd7faf854b30b02e62087a5ac40cd696b4e2c49646f10a08cdcdf3" Nov 02 14:41:26.266187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:26.266111 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fe080d0bbcd7faf854b30b02e62087a5ac40cd696b4e2c49646f10a08cdcdf3"} err="failed to get container status \"4fe080d0bbcd7faf854b30b02e62087a5ac40cd696b4e2c49646f10a08cdcdf3\": rpc error: code = NotFound desc = could not find container \"4fe080d0bbcd7faf854b30b02e62087a5ac40cd696b4e2c49646f10a08cdcdf3\": container with ID starting with 4fe080d0bbcd7faf854b30b02e62087a5ac40cd696b4e2c49646f10a08cdcdf3 not found: ID does not exist" Nov 02 14:41:26.313778 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:26.313741 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z"] Nov 02 14:41:26.325296 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:26.325267 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ba53f-77b6455cc9-6kd9z"] Nov 02 14:41:26.949218 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:26.949186 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8" path="/var/lib/kubelet/pods/dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8/volumes" Nov 02 14:41:30.279874 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.279840 4866 generic.go:356] "Generic (PLEG): container finished" podID="6bc409ca-3338-4474-bad2-aeb03d23aedc" containerID="f305c6bde605dddd1576a2ebfd1a7acb02a443b2c4eeaf6f1ca40ed435648e6c" exitCode=0 Nov 02 14:41:30.280286 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.279889 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" event={"ID":"6bc409ca-3338-4474-bad2-aeb03d23aedc","Type":"ContainerDied","Data":"f305c6bde605dddd1576a2ebfd1a7acb02a443b2c4eeaf6f1ca40ed435648e6c"} Nov 02 14:41:30.280286 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.279939 4866 scope.go:122] "RemoveContainer" containerID="17c24af6f40d321f5fc7021d3fab583d05511ffcc6a4af356fb15f4159f89cd0" Nov 02 14:41:30.383392 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.383362 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:41:30.520789 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.520749 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-data\") pod \"6bc409ca-3338-4474-bad2-aeb03d23aedc\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " Nov 02 14:41:30.521019 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.520799 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-podinfo\") pod \"6bc409ca-3338-4474-bad2-aeb03d23aedc\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " Nov 02 14:41:30.521019 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.520837 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-workload-socket\") pod \"6bc409ca-3338-4474-bad2-aeb03d23aedc\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " Nov 02 14:41:30.521019 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.520907 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-credential-socket\") pod \"6bc409ca-3338-4474-bad2-aeb03d23aedc\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " Nov 02 14:41:30.521019 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.520961 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc409ca-3338-4474-bad2-aeb03d23aedc-kube-api-access-ncpjr\" (UniqueName: \"kubernetes.io/projected/6bc409ca-3338-4474-bad2-aeb03d23aedc-kube-api-access-ncpjr\") pod \"6bc409ca-3338-4474-bad2-aeb03d23aedc\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " Nov 02 14:41:30.521019 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.521017 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-envoy\") pod \"6bc409ca-3338-4474-bad2-aeb03d23aedc\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " Nov 02 14:41:30.521309 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.521054 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-token\" (UniqueName: \"kubernetes.io/projected/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-token\") pod \"6bc409ca-3338-4474-bad2-aeb03d23aedc\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " Nov 02 14:41:30.521309 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.521109 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-data" pod "6bc409ca-3338-4474-bad2-aeb03d23aedc" (UID: "6bc409ca-3338-4474-bad2-aeb03d23aedc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:41:30.521309 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.521122 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-workload-certs\") pod \"6bc409ca-3338-4474-bad2-aeb03d23aedc\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " Nov 02 14:41:30.521309 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.521211 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc409ca-3338-4474-bad2-aeb03d23aedc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc409ca-3338-4474-bad2-aeb03d23aedc-istiod-ca-cert\") pod \"6bc409ca-3338-4474-bad2-aeb03d23aedc\" (UID: \"6bc409ca-3338-4474-bad2-aeb03d23aedc\") " Nov 02 14:41:30.521309 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.521232 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-workload-socket" pod "6bc409ca-3338-4474-bad2-aeb03d23aedc" (UID: "6bc409ca-3338-4474-bad2-aeb03d23aedc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:41:30.521542 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.521396 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-workload-certs" pod "6bc409ca-3338-4474-bad2-aeb03d23aedc" (UID: "6bc409ca-3338-4474-bad2-aeb03d23aedc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:41:30.521542 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.521405 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-credential-socket" pod "6bc409ca-3338-4474-bad2-aeb03d23aedc" (UID: "6bc409ca-3338-4474-bad2-aeb03d23aedc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:41:30.521751 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.521725 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:30.521751 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.521745 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:30.521869 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.521759 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:30.521869 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.521774 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:30.521869 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.521775 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc409ca-3338-4474-bad2-aeb03d23aedc-istiod-ca-cert" pod "6bc409ca-3338-4474-bad2-aeb03d23aedc" (UID: "6bc409ca-3338-4474-bad2-aeb03d23aedc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:41:30.523877 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.523852 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-envoy" pod "6bc409ca-3338-4474-bad2-aeb03d23aedc" (UID: "6bc409ca-3338-4474-bad2-aeb03d23aedc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:41:30.524008 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.523899 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-token" pod "6bc409ca-3338-4474-bad2-aeb03d23aedc" (UID: "6bc409ca-3338-4474-bad2-aeb03d23aedc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:41:30.524008 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.523913 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-podinfo" pod "6bc409ca-3338-4474-bad2-aeb03d23aedc" (UID: "6bc409ca-3338-4474-bad2-aeb03d23aedc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:41:30.524008 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.523921 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc409ca-3338-4474-bad2-aeb03d23aedc-kube-api-access-ncpjr" pod "6bc409ca-3338-4474-bad2-aeb03d23aedc" (UID: "6bc409ca-3338-4474-bad2-aeb03d23aedc"). InnerVolumeSpecName "kube-api-access-ncpjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:41:30.623080 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.623042 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncpjr\" (UniqueName: \"kubernetes.io/projected/6bc409ca-3338-4474-bad2-aeb03d23aedc-kube-api-access-ncpjr\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:30.623080 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.623074 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:30.623080 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.623084 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:30.623325 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.623094 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc409ca-3338-4474-bad2-aeb03d23aedc-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:30.623325 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:30.623102 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc409ca-3338-4474-bad2-aeb03d23aedc-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:31.287350 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:31.287269 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" Nov 02 14:41:31.287350 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:31.287312 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l" event={"ID":"6bc409ca-3338-4474-bad2-aeb03d23aedc","Type":"ContainerDied","Data":"0899899f4117ca3f9a1401023199ab9909d1a49e45cd623e942c1ab58a7c176f"} Nov 02 14:41:31.287857 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:31.287370 4866 scope.go:122] "RemoveContainer" containerID="f305c6bde605dddd1576a2ebfd1a7acb02a443b2c4eeaf6f1ca40ed435648e6c" Nov 02 14:41:31.389835 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:31.389803 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l"] Nov 02 14:41:31.400473 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:31.400442 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5ba53f-openshift-default-6d979b9bdc-slv8l"] Nov 02 14:41:31.693697 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:31.693661 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w"] Nov 02 14:41:31.733692 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:31.733659 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j"] Nov 02 14:41:32.945092 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:32.945058 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc409ca-3338-4474-bad2-aeb03d23aedc" path="/var/lib/kubelet/pods/6bc409ca-3338-4474-bad2-aeb03d23aedc/volumes" Nov 02 14:41:36.009982 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:41:36.009730 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:41:36.694611 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:36.694558 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" podUID="b6e67ad2-0fb8-4025-ac30-8ea39ef94f51" containerName="payload-processing" containerID="cri-o://51c129b40af48f58584ab3674fd94ac0a5ba8659f1a56ab875c894594f7baa67" gracePeriod=25 Nov 02 14:41:36.734839 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:36.734783 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" podUID="7ff352ba-6b9c-4cb7-94d6-efc1de88be37" containerName="payload-pre-processing" containerID="cri-o://9d8da50c6d06f351ab137e68728c9ccba8ae4d5bf8f44fbbb1f7324956a8a698" gracePeriod=25 Nov 02 14:41:37.134353 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:37.134264 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx"] Nov 02 14:41:37.134799 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:37.134537 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" podUID="8d48de97-8d09-4a03-9f59-6422f88891a1" containerName="maas-api" containerID="cri-o://7733af0f6aa359f8aaa79c8a3fc0fb3c25091ae6bf8b6fd106fafda3d3504962" gracePeriod=30 Nov 02 14:41:37.323746 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:37.323714 4866 generic.go:356] "Generic (PLEG): container finished" podID="8d48de97-8d09-4a03-9f59-6422f88891a1" containerID="7733af0f6aa359f8aaa79c8a3fc0fb3c25091ae6bf8b6fd106fafda3d3504962" exitCode=0 Nov 02 14:41:37.323935 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:37.323783 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" event={"ID":"8d48de97-8d09-4a03-9f59-6422f88891a1","Type":"ContainerDied","Data":"7733af0f6aa359f8aaa79c8a3fc0fb3c25091ae6bf8b6fd106fafda3d3504962"} Nov 02 14:41:37.395390 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:37.395316 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" Nov 02 14:41:37.494200 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:37.494170 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d48de97-8d09-4a03-9f59-6422f88891a1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d48de97-8d09-4a03-9f59-6422f88891a1-openshift-service-ca\") pod \"8d48de97-8d09-4a03-9f59-6422f88891a1\" (UID: \"8d48de97-8d09-4a03-9f59-6422f88891a1\") " Nov 02 14:41:37.494395 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:37.494255 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d48de97-8d09-4a03-9f59-6422f88891a1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d48de97-8d09-4a03-9f59-6422f88891a1-maas-api-tls\") pod \"8d48de97-8d09-4a03-9f59-6422f88891a1\" (UID: \"8d48de97-8d09-4a03-9f59-6422f88891a1\") " Nov 02 14:41:37.494395 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:37.494340 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d48de97-8d09-4a03-9f59-6422f88891a1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d48de97-8d09-4a03-9f59-6422f88891a1-metrics-certs\") pod \"8d48de97-8d09-4a03-9f59-6422f88891a1\" (UID: \"8d48de97-8d09-4a03-9f59-6422f88891a1\") " Nov 02 14:41:37.494395 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:37.494384 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d48de97-8d09-4a03-9f59-6422f88891a1-kube-api-access-545lt\" (UniqueName: \"kubernetes.io/projected/8d48de97-8d09-4a03-9f59-6422f88891a1-kube-api-access-545lt\") pod \"8d48de97-8d09-4a03-9f59-6422f88891a1\" (UID: \"8d48de97-8d09-4a03-9f59-6422f88891a1\") " Nov 02 14:41:37.494577 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:37.494559 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d48de97-8d09-4a03-9f59-6422f88891a1-openshift-service-ca" pod "8d48de97-8d09-4a03-9f59-6422f88891a1" (UID: "8d48de97-8d09-4a03-9f59-6422f88891a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:41:37.496432 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:37.496405 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d48de97-8d09-4a03-9f59-6422f88891a1-metrics-certs" pod "8d48de97-8d09-4a03-9f59-6422f88891a1" (UID: "8d48de97-8d09-4a03-9f59-6422f88891a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:41:37.496549 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:37.496499 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d48de97-8d09-4a03-9f59-6422f88891a1-maas-api-tls" pod "8d48de97-8d09-4a03-9f59-6422f88891a1" (UID: "8d48de97-8d09-4a03-9f59-6422f88891a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:41:37.496549 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:37.496526 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d48de97-8d09-4a03-9f59-6422f88891a1-kube-api-access-545lt" pod "8d48de97-8d09-4a03-9f59-6422f88891a1" (UID: "8d48de97-8d09-4a03-9f59-6422f88891a1"). InnerVolumeSpecName "kube-api-access-545lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:41:37.596342 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:37.596300 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-545lt\" (UniqueName: \"kubernetes.io/projected/8d48de97-8d09-4a03-9f59-6422f88891a1-kube-api-access-545lt\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:37.596342 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:37.596341 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d48de97-8d09-4a03-9f59-6422f88891a1-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:37.596562 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:37.596357 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d48de97-8d09-4a03-9f59-6422f88891a1-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:37.596562 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:37.596373 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d48de97-8d09-4a03-9f59-6422f88891a1-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:38.338555 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:38.338516 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" event={"ID":"8d48de97-8d09-4a03-9f59-6422f88891a1","Type":"ContainerDied","Data":"e139187f3690a27c660235e2ece8369f9471a9797236c98abe75b049381c7350"} Nov 02 14:41:38.339056 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:38.338569 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx" Nov 02 14:41:38.339056 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:38.338601 4866 scope.go:122] "RemoveContainer" containerID="7733af0f6aa359f8aaa79c8a3fc0fb3c25091ae6bf8b6fd106fafda3d3504962" Nov 02 14:41:38.390630 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:38.390590 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx"] Nov 02 14:41:38.402574 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:38.402537 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-944506-658755674f-7mghx"] Nov 02 14:41:38.945013 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:38.944977 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d48de97-8d09-4a03-9f59-6422f88891a1" path="/var/lib/kubelet/pods/8d48de97-8d09-4a03-9f59-6422f88891a1/volumes" Nov 02 14:41:46.318879 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:46.318835 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" podUID="b6e67ad2-0fb8-4025-ac30-8ea39ef94f51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:41:46.319334 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:46.318835 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" podUID="7ff352ba-6b9c-4cb7-94d6-efc1de88be37" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:41:56.319073 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:56.319029 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" podUID="b6e67ad2-0fb8-4025-ac30-8ea39ef94f51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:41:56.319487 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:56.319072 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" podUID="7ff352ba-6b9c-4cb7-94d6-efc1de88be37" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:41:57.002025 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.001998 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" Nov 02 14:41:57.030010 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.029980 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" Nov 02 14:41:57.098467 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.098432 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6e67ad2-0fb8-4025-ac30-8ea39ef94f51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e67ad2-0fb8-4025-ac30-8ea39ef94f51-plugins-config-volume\") pod \"b6e67ad2-0fb8-4025-ac30-8ea39ef94f51\" (UID: \"b6e67ad2-0fb8-4025-ac30-8ea39ef94f51\") " Nov 02 14:41:57.098677 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.098650 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff352ba-6b9c-4cb7-94d6-efc1de88be37-kube-api-access-lh7cl\" (UniqueName: \"kubernetes.io/projected/7ff352ba-6b9c-4cb7-94d6-efc1de88be37-kube-api-access-lh7cl\") pod \"7ff352ba-6b9c-4cb7-94d6-efc1de88be37\" (UID: \"7ff352ba-6b9c-4cb7-94d6-efc1de88be37\") " Nov 02 14:41:57.098834 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.098700 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e67ad2-0fb8-4025-ac30-8ea39ef94f51-kube-api-access-dxfzv\" (UniqueName: \"kubernetes.io/projected/b6e67ad2-0fb8-4025-ac30-8ea39ef94f51-kube-api-access-dxfzv\") pod \"b6e67ad2-0fb8-4025-ac30-8ea39ef94f51\" (UID: \"b6e67ad2-0fb8-4025-ac30-8ea39ef94f51\") " Nov 02 14:41:57.098834 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.098744 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ff352ba-6b9c-4cb7-94d6-efc1de88be37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff352ba-6b9c-4cb7-94d6-efc1de88be37-plugins-config-volume\") pod \"7ff352ba-6b9c-4cb7-94d6-efc1de88be37\" (UID: \"7ff352ba-6b9c-4cb7-94d6-efc1de88be37\") " Nov 02 14:41:57.098986 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.098906 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e67ad2-0fb8-4025-ac30-8ea39ef94f51-plugins-config-volume" pod "b6e67ad2-0fb8-4025-ac30-8ea39ef94f51" (UID: "b6e67ad2-0fb8-4025-ac30-8ea39ef94f51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:41:57.099335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.099309 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e67ad2-0fb8-4025-ac30-8ea39ef94f51-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:57.099335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.099325 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff352ba-6b9c-4cb7-94d6-efc1de88be37-plugins-config-volume" pod "7ff352ba-6b9c-4cb7-94d6-efc1de88be37" (UID: "7ff352ba-6b9c-4cb7-94d6-efc1de88be37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:41:57.100832 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.100809 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e67ad2-0fb8-4025-ac30-8ea39ef94f51-kube-api-access-dxfzv" pod "b6e67ad2-0fb8-4025-ac30-8ea39ef94f51" (UID: "b6e67ad2-0fb8-4025-ac30-8ea39ef94f51"). InnerVolumeSpecName "kube-api-access-dxfzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:41:57.101483 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.101461 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff352ba-6b9c-4cb7-94d6-efc1de88be37-kube-api-access-lh7cl" pod "7ff352ba-6b9c-4cb7-94d6-efc1de88be37" (UID: "7ff352ba-6b9c-4cb7-94d6-efc1de88be37"). InnerVolumeSpecName "kube-api-access-lh7cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:41:57.200384 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.200351 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh7cl\" (UniqueName: \"kubernetes.io/projected/7ff352ba-6b9c-4cb7-94d6-efc1de88be37-kube-api-access-lh7cl\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:57.200384 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.200378 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxfzv\" (UniqueName: \"kubernetes.io/projected/b6e67ad2-0fb8-4025-ac30-8ea39ef94f51-kube-api-access-dxfzv\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:57.200592 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.200390 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff352ba-6b9c-4cb7-94d6-efc1de88be37-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:41:57.432557 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.432475 4866 generic.go:356] "Generic (PLEG): container finished" podID="b6e67ad2-0fb8-4025-ac30-8ea39ef94f51" containerID="51c129b40af48f58584ab3674fd94ac0a5ba8659f1a56ab875c894594f7baa67" exitCode=0 Nov 02 14:41:57.432557 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.432519 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" Nov 02 14:41:57.433115 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.432554 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" event={"ID":"b6e67ad2-0fb8-4025-ac30-8ea39ef94f51","Type":"ContainerDied","Data":"51c129b40af48f58584ab3674fd94ac0a5ba8659f1a56ab875c894594f7baa67"} Nov 02 14:41:57.433115 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.432608 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w" event={"ID":"b6e67ad2-0fb8-4025-ac30-8ea39ef94f51","Type":"ContainerDied","Data":"6b86d00ee0b5a7a55fdbb67380bbad454da4a4104c8a98e0716d856a2b5245a0"} Nov 02 14:41:57.433115 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.432640 4866 scope.go:122] "RemoveContainer" containerID="51c129b40af48f58584ab3674fd94ac0a5ba8659f1a56ab875c894594f7baa67" Nov 02 14:41:57.434041 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.434008 4866 generic.go:356] "Generic (PLEG): container finished" podID="7ff352ba-6b9c-4cb7-94d6-efc1de88be37" containerID="9d8da50c6d06f351ab137e68728c9ccba8ae4d5bf8f44fbbb1f7324956a8a698" exitCode=0 Nov 02 14:41:57.434150 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.434056 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" Nov 02 14:41:57.434150 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.434072 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" event={"ID":"7ff352ba-6b9c-4cb7-94d6-efc1de88be37","Type":"ContainerDied","Data":"9d8da50c6d06f351ab137e68728c9ccba8ae4d5bf8f44fbbb1f7324956a8a698"} Nov 02 14:41:57.434150 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.434113 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j" event={"ID":"7ff352ba-6b9c-4cb7-94d6-efc1de88be37","Type":"ContainerDied","Data":"df04d372aa99d348552eced2cc31804908f7c05c7eca61a0c6bf119248825fed"} Nov 02 14:41:57.442667 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.442647 4866 scope.go:122] "RemoveContainer" containerID="51c129b40af48f58584ab3674fd94ac0a5ba8659f1a56ab875c894594f7baa67" Nov 02 14:41:57.443023 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:41:57.442999 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51c129b40af48f58584ab3674fd94ac0a5ba8659f1a56ab875c894594f7baa67\": container with ID starting with 51c129b40af48f58584ab3674fd94ac0a5ba8659f1a56ab875c894594f7baa67 not found: ID does not exist" containerID="51c129b40af48f58584ab3674fd94ac0a5ba8659f1a56ab875c894594f7baa67" Nov 02 14:41:57.443084 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.443030 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c129b40af48f58584ab3674fd94ac0a5ba8659f1a56ab875c894594f7baa67"} err="failed to get container status \"51c129b40af48f58584ab3674fd94ac0a5ba8659f1a56ab875c894594f7baa67\": rpc error: code = NotFound desc = could not find container \"51c129b40af48f58584ab3674fd94ac0a5ba8659f1a56ab875c894594f7baa67\": container with ID starting with 51c129b40af48f58584ab3674fd94ac0a5ba8659f1a56ab875c894594f7baa67 not found: ID does not exist" Nov 02 14:41:57.443084 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.443049 4866 scope.go:122] "RemoveContainer" containerID="9d8da50c6d06f351ab137e68728c9ccba8ae4d5bf8f44fbbb1f7324956a8a698" Nov 02 14:41:57.451858 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.451841 4866 scope.go:122] "RemoveContainer" containerID="9d8da50c6d06f351ab137e68728c9ccba8ae4d5bf8f44fbbb1f7324956a8a698" Nov 02 14:41:57.452170 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:41:57.452151 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d8da50c6d06f351ab137e68728c9ccba8ae4d5bf8f44fbbb1f7324956a8a698\": container with ID starting with 9d8da50c6d06f351ab137e68728c9ccba8ae4d5bf8f44fbbb1f7324956a8a698 not found: ID does not exist" containerID="9d8da50c6d06f351ab137e68728c9ccba8ae4d5bf8f44fbbb1f7324956a8a698" Nov 02 14:41:57.452236 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.452177 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d8da50c6d06f351ab137e68728c9ccba8ae4d5bf8f44fbbb1f7324956a8a698"} err="failed to get container status \"9d8da50c6d06f351ab137e68728c9ccba8ae4d5bf8f44fbbb1f7324956a8a698\": rpc error: code = NotFound desc = could not find container \"9d8da50c6d06f351ab137e68728c9ccba8ae4d5bf8f44fbbb1f7324956a8a698\": container with ID starting with 9d8da50c6d06f351ab137e68728c9ccba8ae4d5bf8f44fbbb1f7324956a8a698 not found: ID does not exist" Nov 02 14:41:57.489926 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.489896 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w"] Nov 02 14:41:57.502539 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.502503 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-944506-5bcfb5fb67-8cq9w"] Nov 02 14:41:57.537532 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.537497 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j"] Nov 02 14:41:57.542848 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:57.542816 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-944506-786f89d5dc-tmm5j"] Nov 02 14:41:58.944806 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:58.944772 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ff352ba-6b9c-4cb7-94d6-efc1de88be37" path="/var/lib/kubelet/pods/7ff352ba-6b9c-4cb7-94d6-efc1de88be37/volumes" Nov 02 14:41:58.945227 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:41:58.945212 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6e67ad2-0fb8-4025-ac30-8ea39ef94f51" path="/var/lib/kubelet/pods/b6e67ad2-0fb8-4025-ac30-8ea39ef94f51/volumes" Nov 02 14:42:06.009469 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:42:06.009424 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:42:14.282079 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:14.282041 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6"] Nov 02 14:42:15.575191 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:15.575159 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:42:15.595185 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:15.595154 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:42:19.283070 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:19.282992 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6" podUID="29bfcefb-2523-4638-8807-f52e7eb4e86f" containerName="payload-processing" containerID="cri-o://82b8dddce3b5191e689286b894ef05d0aac0f369594311a4a97b7bd4a3d6fd85" gracePeriod=25 Nov 02 14:42:19.549569 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:19.549495 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6" Nov 02 14:42:19.557281 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:19.557246 4866 generic.go:356] "Generic (PLEG): container finished" podID="29bfcefb-2523-4638-8807-f52e7eb4e86f" containerID="82b8dddce3b5191e689286b894ef05d0aac0f369594311a4a97b7bd4a3d6fd85" exitCode=0 Nov 02 14:42:19.557455 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:19.557299 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6" Nov 02 14:42:19.557455 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:19.557330 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6" event={"ID":"29bfcefb-2523-4638-8807-f52e7eb4e86f","Type":"ContainerDied","Data":"82b8dddce3b5191e689286b894ef05d0aac0f369594311a4a97b7bd4a3d6fd85"} Nov 02 14:42:19.557455 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:19.557387 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6" event={"ID":"29bfcefb-2523-4638-8807-f52e7eb4e86f","Type":"ContainerDied","Data":"7e157c70175dcf22188830cc8f4d8e1ded85746e04922e7bb4a3feb55baa5db9"} Nov 02 14:42:19.557455 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:19.557414 4866 scope.go:122] "RemoveContainer" containerID="82b8dddce3b5191e689286b894ef05d0aac0f369594311a4a97b7bd4a3d6fd85" Nov 02 14:42:19.567219 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:19.567197 4866 scope.go:122] "RemoveContainer" containerID="82b8dddce3b5191e689286b894ef05d0aac0f369594311a4a97b7bd4a3d6fd85" Nov 02 14:42:19.567576 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:42:19.567546 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82b8dddce3b5191e689286b894ef05d0aac0f369594311a4a97b7bd4a3d6fd85\": container with ID starting with 82b8dddce3b5191e689286b894ef05d0aac0f369594311a4a97b7bd4a3d6fd85 not found: ID does not exist" containerID="82b8dddce3b5191e689286b894ef05d0aac0f369594311a4a97b7bd4a3d6fd85" Nov 02 14:42:19.567696 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:19.567583 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b8dddce3b5191e689286b894ef05d0aac0f369594311a4a97b7bd4a3d6fd85"} err="failed to get container status \"82b8dddce3b5191e689286b894ef05d0aac0f369594311a4a97b7bd4a3d6fd85\": rpc error: code = NotFound desc = could not find container \"82b8dddce3b5191e689286b894ef05d0aac0f369594311a4a97b7bd4a3d6fd85\": container with ID starting with 82b8dddce3b5191e689286b894ef05d0aac0f369594311a4a97b7bd4a3d6fd85 not found: ID does not exist" Nov 02 14:42:19.733511 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:19.733472 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29bfcefb-2523-4638-8807-f52e7eb4e86f-kube-api-access-ch9qc\" (UniqueName: \"kubernetes.io/projected/29bfcefb-2523-4638-8807-f52e7eb4e86f-kube-api-access-ch9qc\") pod \"29bfcefb-2523-4638-8807-f52e7eb4e86f\" (UID: \"29bfcefb-2523-4638-8807-f52e7eb4e86f\") " Nov 02 14:42:19.733511 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:19.733521 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29bfcefb-2523-4638-8807-f52e7eb4e86f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bfcefb-2523-4638-8807-f52e7eb4e86f-plugins-config-volume\") pod \"29bfcefb-2523-4638-8807-f52e7eb4e86f\" (UID: \"29bfcefb-2523-4638-8807-f52e7eb4e86f\") " Nov 02 14:42:19.733898 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:19.733870 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29bfcefb-2523-4638-8807-f52e7eb4e86f-plugins-config-volume" pod "29bfcefb-2523-4638-8807-f52e7eb4e86f" (UID: "29bfcefb-2523-4638-8807-f52e7eb4e86f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:42:19.735629 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:19.735608 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29bfcefb-2523-4638-8807-f52e7eb4e86f-kube-api-access-ch9qc" pod "29bfcefb-2523-4638-8807-f52e7eb4e86f" (UID: "29bfcefb-2523-4638-8807-f52e7eb4e86f"). InnerVolumeSpecName "kube-api-access-ch9qc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:19.834883 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:19.834797 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch9qc\" (UniqueName: \"kubernetes.io/projected/29bfcefb-2523-4638-8807-f52e7eb4e86f-kube-api-access-ch9qc\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:19.834883 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:19.834833 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bfcefb-2523-4638-8807-f52e7eb4e86f-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:19.903638 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:19.903596 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6"] Nov 02 14:42:19.907895 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:19.907862 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f72bc6-9b78d7d7c-2mbq6"] Nov 02 14:42:20.947071 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:20.947039 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29bfcefb-2523-4638-8807-f52e7eb4e86f" path="/var/lib/kubelet/pods/29bfcefb-2523-4638-8807-f52e7eb4e86f/volumes" Nov 02 14:42:30.051802 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:30.051767 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt"] Nov 02 14:42:30.626484 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:30.626423 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" podUID="b2a9fbab-fdf4-4d57-9aae-74842fb231c9" containerName="maas-api" containerID="cri-o://3436d2d525ce5d2a76ab378ae2f724666587865e96b2ff284292cc4bdbb6b1a1" gracePeriod=30 Nov 02 14:42:30.884042 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:30.883975 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" Nov 02 14:42:30.958154 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:30.958116 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-maas-api-tls\") pod \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\" (UID: \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\") " Nov 02 14:42:30.958358 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:30.958269 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-kube-api-access-m5pxx\" (UniqueName: \"kubernetes.io/projected/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-kube-api-access-m5pxx\") pod \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\" (UID: \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\") " Nov 02 14:42:30.958358 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:30.958329 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-metrics-certs\") pod \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\" (UID: \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\") " Nov 02 14:42:30.958513 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:30.958378 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-openshift-service-ca\") pod \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\" (UID: \"b2a9fbab-fdf4-4d57-9aae-74842fb231c9\") " Nov 02 14:42:30.959014 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:30.958984 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-openshift-service-ca" pod "b2a9fbab-fdf4-4d57-9aae-74842fb231c9" (UID: "b2a9fbab-fdf4-4d57-9aae-74842fb231c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:42:30.960509 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:30.960480 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-metrics-certs" pod "b2a9fbab-fdf4-4d57-9aae-74842fb231c9" (UID: "b2a9fbab-fdf4-4d57-9aae-74842fb231c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:42:30.960768 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:30.960739 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-maas-api-tls" pod "b2a9fbab-fdf4-4d57-9aae-74842fb231c9" (UID: "b2a9fbab-fdf4-4d57-9aae-74842fb231c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:42:30.960768 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:30.960758 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-kube-api-access-m5pxx" pod "b2a9fbab-fdf4-4d57-9aae-74842fb231c9" (UID: "b2a9fbab-fdf4-4d57-9aae-74842fb231c9"). InnerVolumeSpecName "kube-api-access-m5pxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:31.060138 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.060102 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5pxx\" (UniqueName: \"kubernetes.io/projected/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-kube-api-access-m5pxx\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:31.060138 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.060134 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:31.060138 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.060143 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:31.060645 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.060152 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a9fbab-fdf4-4d57-9aae-74842fb231c9-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:31.249453 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249418 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf"] Nov 02 14:42:31.249831 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249818 4866 cpu_manager.go:414] "containerMap: removing container" podUID="2d243c72-dc2f-458e-8f7b-23f89144e7bf" containerName="payload-pre-processing" Nov 02 14:42:31.249881 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249832 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d243c72-dc2f-458e-8f7b-23f89144e7bf" containerName="payload-pre-processing" Nov 02 14:42:31.249881 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249847 4866 cpu_manager.go:414] "containerMap: removing container" podUID="dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8" containerName="payload-pre-processing" Nov 02 14:42:31.249881 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249854 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8" containerName="payload-pre-processing" Nov 02 14:42:31.249881 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249864 4866 cpu_manager.go:414] "containerMap: removing container" podUID="b6e67ad2-0fb8-4025-ac30-8ea39ef94f51" containerName="payload-processing" Nov 02 14:42:31.249881 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249869 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6e67ad2-0fb8-4025-ac30-8ea39ef94f51" containerName="payload-processing" Nov 02 14:42:31.249881 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249879 4866 cpu_manager.go:414] "containerMap: removing container" podUID="3d5b2d9a-c9e4-4e9c-b080-7056e2016709" containerName="main" Nov 02 14:42:31.249881 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249883 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5b2d9a-c9e4-4e9c-b080-7056e2016709" containerName="main" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249890 4866 cpu_manager.go:414] "containerMap: removing container" podUID="7ff352ba-6b9c-4cb7-94d6-efc1de88be37" containerName="payload-pre-processing" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249894 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff352ba-6b9c-4cb7-94d6-efc1de88be37" containerName="payload-pre-processing" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249902 4866 cpu_manager.go:414] "containerMap: removing container" podUID="29dea1dd-e792-415a-9c82-dfa2edf71510" containerName="main" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249907 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29dea1dd-e792-415a-9c82-dfa2edf71510" containerName="main" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249915 4866 cpu_manager.go:414] "containerMap: removing container" podUID="6803eccc-186b-4560-9dab-82bee489a940" containerName="maas-api" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249920 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6803eccc-186b-4560-9dab-82bee489a940" containerName="maas-api" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249937 4866 cpu_manager.go:414] "containerMap: removing container" podUID="6bc409ca-3338-4474-bad2-aeb03d23aedc" containerName="istio-proxy" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249942 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc409ca-3338-4474-bad2-aeb03d23aedc" containerName="istio-proxy" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249965 4866 cpu_manager.go:414] "containerMap: removing container" podUID="29bfcefb-2523-4638-8807-f52e7eb4e86f" containerName="payload-processing" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249970 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29bfcefb-2523-4638-8807-f52e7eb4e86f" containerName="payload-processing" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249976 4866 cpu_manager.go:414] "containerMap: removing container" podUID="6bc409ca-3338-4474-bad2-aeb03d23aedc" containerName="istio-proxy" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249980 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc409ca-3338-4474-bad2-aeb03d23aedc" containerName="istio-proxy" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249989 4866 cpu_manager.go:414] "containerMap: removing container" podUID="8d48de97-8d09-4a03-9f59-6422f88891a1" containerName="maas-api" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.249993 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d48de97-8d09-4a03-9f59-6422f88891a1" containerName="maas-api" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.250001 4866 cpu_manager.go:414] "containerMap: removing container" podUID="b2a9fbab-fdf4-4d57-9aae-74842fb231c9" containerName="maas-api" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.250006 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a9fbab-fdf4-4d57-9aae-74842fb231c9" containerName="maas-api" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.250069 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d243c72-dc2f-458e-8f7b-23f89144e7bf" containerName="payload-pre-processing" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.250079 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd2bd27f-ce10-41d9-9c35-ff8b6995c6a8" containerName="payload-pre-processing" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.250084 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6e67ad2-0fb8-4025-ac30-8ea39ef94f51" containerName="payload-processing" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.250089 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5b2d9a-c9e4-4e9c-b080-7056e2016709" containerName="main" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.250095 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="29dea1dd-e792-415a-9c82-dfa2edf71510" containerName="main" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.250100 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff352ba-6b9c-4cb7-94d6-efc1de88be37" containerName="payload-pre-processing" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.250105 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="6803eccc-186b-4560-9dab-82bee489a940" containerName="maas-api" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.250112 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc409ca-3338-4474-bad2-aeb03d23aedc" containerName="istio-proxy" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.250117 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="29bfcefb-2523-4638-8807-f52e7eb4e86f" containerName="payload-processing" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.250123 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc409ca-3338-4474-bad2-aeb03d23aedc" containerName="istio-proxy" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.250129 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d48de97-8d09-4a03-9f59-6422f88891a1" containerName="maas-api" Nov 02 14:42:31.250140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.250136 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a9fbab-fdf4-4d57-9aae-74842fb231c9" containerName="maas-api" Nov 02 14:42:31.253471 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.253453 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.295185 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.295148 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf"] Nov 02 14:42:31.363155 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.363106 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5028a6-bf31-4b46-8252-caf390803e5e-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.363155 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.363156 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.363440 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.363277 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.363440 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.363312 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzf5p\" (UniqueName: \"kubernetes.io/projected/0a5028a6-bf31-4b46-8252-caf390803e5e-kube-api-access-jzf5p\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.363440 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.363388 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.363440 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.363427 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.363655 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.363449 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.363655 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.363467 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.363655 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.363598 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.464936 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.464896 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5028a6-bf31-4b46-8252-caf390803e5e-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.465167 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.464974 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.465167 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.465027 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.465167 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.465055 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzf5p\" (UniqueName: \"kubernetes.io/projected/0a5028a6-bf31-4b46-8252-caf390803e5e-kube-api-access-jzf5p\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.465167 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.465089 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.465167 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.465135 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.465435 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.465167 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.465435 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.465195 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.465435 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.465254 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.465603 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.465521 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.465719 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.465695 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.465818 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.465781 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5028a6-bf31-4b46-8252-caf390803e5e-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.465818 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.465799 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.465994 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.465976 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.467413 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.467376 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.467980 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.467938 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.491045 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.491011 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzf5p\" (UniqueName: \"kubernetes.io/projected/0a5028a6-bf31-4b46-8252-caf390803e5e-kube-api-access-jzf5p\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.491503 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.491483 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.566493 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.566396 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:31.637986 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.637718 4866 generic.go:356] "Generic (PLEG): container finished" podID="b2a9fbab-fdf4-4d57-9aae-74842fb231c9" containerID="3436d2d525ce5d2a76ab378ae2f724666587865e96b2ff284292cc4bdbb6b1a1" exitCode=0 Nov 02 14:42:31.637986 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.637769 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" Nov 02 14:42:31.637986 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.637861 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" event={"ID":"b2a9fbab-fdf4-4d57-9aae-74842fb231c9","Type":"ContainerDied","Data":"3436d2d525ce5d2a76ab378ae2f724666587865e96b2ff284292cc4bdbb6b1a1"} Nov 02 14:42:31.637986 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.637910 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt" event={"ID":"b2a9fbab-fdf4-4d57-9aae-74842fb231c9","Type":"ContainerDied","Data":"e54b93f7be1a517a1037b75d1531c7117667614c22bdc7a9e16e03b006f1fc67"} Nov 02 14:42:31.637986 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.637940 4866 scope.go:122] "RemoveContainer" containerID="3436d2d525ce5d2a76ab378ae2f724666587865e96b2ff284292cc4bdbb6b1a1" Nov 02 14:42:31.656720 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.656692 4866 scope.go:122] "RemoveContainer" containerID="3436d2d525ce5d2a76ab378ae2f724666587865e96b2ff284292cc4bdbb6b1a1" Nov 02 14:42:31.657119 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:42:31.657090 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3436d2d525ce5d2a76ab378ae2f724666587865e96b2ff284292cc4bdbb6b1a1\": container with ID starting with 3436d2d525ce5d2a76ab378ae2f724666587865e96b2ff284292cc4bdbb6b1a1 not found: ID does not exist" containerID="3436d2d525ce5d2a76ab378ae2f724666587865e96b2ff284292cc4bdbb6b1a1" Nov 02 14:42:31.657218 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.657133 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3436d2d525ce5d2a76ab378ae2f724666587865e96b2ff284292cc4bdbb6b1a1"} err="failed to get container status \"3436d2d525ce5d2a76ab378ae2f724666587865e96b2ff284292cc4bdbb6b1a1\": rpc error: code = NotFound desc = could not find container \"3436d2d525ce5d2a76ab378ae2f724666587865e96b2ff284292cc4bdbb6b1a1\": container with ID starting with 3436d2d525ce5d2a76ab378ae2f724666587865e96b2ff284292cc4bdbb6b1a1 not found: ID does not exist" Nov 02 14:42:31.701236 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.701200 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt"] Nov 02 14:42:31.705901 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.705869 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f72bc6-5df5cd74d5-kx8qt"] Nov 02 14:42:31.907495 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.907454 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf"] Nov 02 14:42:31.909141 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:42:31.909110 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5028a6_bf31_4b46_8252_caf390803e5e.slice/crio-4075590fb4b9cf19fce7a1d6d902a0689eaae163d7a5c74b1317f603e84661f5 WatchSource:0}: Error finding container 4075590fb4b9cf19fce7a1d6d902a0689eaae163d7a5c74b1317f603e84661f5: Status 404 returned error can't find the container with id 4075590fb4b9cf19fce7a1d6d902a0689eaae163d7a5c74b1317f603e84661f5 Nov 02 14:42:31.911346 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.911313 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:42:31.911457 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.911377 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:42:31.911457 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:31.911409 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:42:32.645190 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:32.645148 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" event={"ID":"0a5028a6-bf31-4b46-8252-caf390803e5e","Type":"ContainerStarted","Data":"15f7c0a04889c42a4ad240c2072e61bfbc58cb3e6e6fa03e9b19360468681368"} Nov 02 14:42:32.645689 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:32.645205 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" event={"ID":"0a5028a6-bf31-4b46-8252-caf390803e5e","Type":"ContainerStarted","Data":"4075590fb4b9cf19fce7a1d6d902a0689eaae163d7a5c74b1317f603e84661f5"} Nov 02 14:42:32.697541 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:32.697487 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" podStartSLOduration=1.697474506 podStartE2EDuration="1.697474506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:42:32.696939664 +0000 UTC m=+2717.880787002" watchObservedRunningTime="2026-11-02 14:42:32.697474506 +0000 UTC m=+2717.881321843" Nov 02 14:42:32.945889 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:32.945851 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2a9fbab-fdf4-4d57-9aae-74842fb231c9" path="/var/lib/kubelet/pods/b2a9fbab-fdf4-4d57-9aae-74842fb231c9/volumes" Nov 02 14:42:33.567270 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:33.567234 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:33.568701 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:33.568672 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 02 14:42:33.568848 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:33.568722 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" podUID="0a5028a6-bf31-4b46-8252-caf390803e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 02 14:42:34.567622 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:34.567587 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 02 14:42:34.568083 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:34.567644 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" podUID="0a5028a6-bf31-4b46-8252-caf390803e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 02 14:42:35.567382 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:35.567347 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 02 14:42:35.567589 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:35.567399 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" podUID="0a5028a6-bf31-4b46-8252-caf390803e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 02 14:42:36.032800 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:42:36.032762 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:42:36.566680 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:36.566639 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 02 14:42:36.566870 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:36.566702 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" podUID="0a5028a6-bf31-4b46-8252-caf390803e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 02 14:42:37.567495 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:37.567454 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 02 14:42:37.567934 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:37.567504 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" podUID="0a5028a6-bf31-4b46-8252-caf390803e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 02 14:42:38.567580 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:38.567543 4866 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 02 14:42:38.568029 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:38.567606 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" podUID="0a5028a6-bf31-4b46-8252-caf390803e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 02 14:42:39.636580 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:39.636534 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:39.637088 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:39.637064 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:39.637870 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:39.637837 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:43.783508 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:43.783469 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf"] Nov 02 14:42:43.784111 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:43.783749 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" podUID="0a5028a6-bf31-4b46-8252-caf390803e5e" containerName="istio-proxy" containerID="cri-o://15f7c0a04889c42a4ad240c2072e61bfbc58cb3e6e6fa03e9b19360468681368" gracePeriod=30 Nov 02 14:42:43.793957 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:43.793917 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr"] Nov 02 14:42:43.794163 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:43.794133 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" podUID="d94980c9-4b40-448d-9e6d-1a8176035b8e" containerName="istio-proxy" containerID="cri-o://80d396d4def9423894c9004e8368850ac3c97cfce9ed341945f07513243dd985" gracePeriod=30 Nov 02 14:42:49.057748 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.057667 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:49.094273 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.094247 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:42:49.179479 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.179444 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5028a6-bf31-4b46-8252-caf390803e5e-kube-api-access-jzf5p\" (UniqueName: \"kubernetes.io/projected/0a5028a6-bf31-4b46-8252-caf390803e5e-kube-api-access-jzf5p\") pod \"0a5028a6-bf31-4b46-8252-caf390803e5e\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " Nov 02 14:42:49.179653 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.179508 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-token\" (UniqueName: \"kubernetes.io/projected/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-token\") pod \"0a5028a6-bf31-4b46-8252-caf390803e5e\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " Nov 02 14:42:49.179653 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.179546 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-workload-socket\") pod \"0a5028a6-bf31-4b46-8252-caf390803e5e\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " Nov 02 14:42:49.179653 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.179573 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-podinfo\") pod \"0a5028a6-bf31-4b46-8252-caf390803e5e\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " Nov 02 14:42:49.179653 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.179601 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-data\") pod \"0a5028a6-bf31-4b46-8252-caf390803e5e\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " Nov 02 14:42:49.179884 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.179654 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-envoy\") pod \"0a5028a6-bf31-4b46-8252-caf390803e5e\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " Nov 02 14:42:49.179884 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.179746 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-credential-socket\") pod \"0a5028a6-bf31-4b46-8252-caf390803e5e\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " Nov 02 14:42:49.179884 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.179836 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a5028a6-bf31-4b46-8252-caf390803e5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5028a6-bf31-4b46-8252-caf390803e5e-istiod-ca-cert\") pod \"0a5028a6-bf31-4b46-8252-caf390803e5e\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " Nov 02 14:42:49.180094 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.179896 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-workload-socket" pod "0a5028a6-bf31-4b46-8252-caf390803e5e" (UID: "0a5028a6-bf31-4b46-8252-caf390803e5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.180094 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.179920 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-workload-certs\") pod \"0a5028a6-bf31-4b46-8252-caf390803e5e\" (UID: \"0a5028a6-bf31-4b46-8252-caf390803e5e\") " Nov 02 14:42:49.180345 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.180316 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.180345 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.179909 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-data" pod "0a5028a6-bf31-4b46-8252-caf390803e5e" (UID: "0a5028a6-bf31-4b46-8252-caf390803e5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.180552 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.180131 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-credential-socket" pod "0a5028a6-bf31-4b46-8252-caf390803e5e" (UID: "0a5028a6-bf31-4b46-8252-caf390803e5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.180552 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.180246 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5028a6-bf31-4b46-8252-caf390803e5e-istiod-ca-cert" pod "0a5028a6-bf31-4b46-8252-caf390803e5e" (UID: "0a5028a6-bf31-4b46-8252-caf390803e5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:42:49.180552 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.180317 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-workload-certs" pod "0a5028a6-bf31-4b46-8252-caf390803e5e" (UID: "0a5028a6-bf31-4b46-8252-caf390803e5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.182083 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.182058 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-envoy" pod "0a5028a6-bf31-4b46-8252-caf390803e5e" (UID: "0a5028a6-bf31-4b46-8252-caf390803e5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.182278 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.182253 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-token" pod "0a5028a6-bf31-4b46-8252-caf390803e5e" (UID: "0a5028a6-bf31-4b46-8252-caf390803e5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:49.182358 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.182262 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5028a6-bf31-4b46-8252-caf390803e5e-kube-api-access-jzf5p" pod "0a5028a6-bf31-4b46-8252-caf390803e5e" (UID: "0a5028a6-bf31-4b46-8252-caf390803e5e"). InnerVolumeSpecName "kube-api-access-jzf5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:49.182834 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.182816 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-podinfo" pod "0a5028a6-bf31-4b46-8252-caf390803e5e" (UID: "0a5028a6-bf31-4b46-8252-caf390803e5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:42:49.281002 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.280965 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-workload-certs\") pod \"d94980c9-4b40-448d-9e6d-1a8176035b8e\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " Nov 02 14:42:49.281181 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281034 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-envoy\") pod \"d94980c9-4b40-448d-9e6d-1a8176035b8e\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " Nov 02 14:42:49.281181 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281060 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-token\" (UniqueName: \"kubernetes.io/projected/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-token\") pod \"d94980c9-4b40-448d-9e6d-1a8176035b8e\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " Nov 02 14:42:49.281181 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281097 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-credential-socket\") pod \"d94980c9-4b40-448d-9e6d-1a8176035b8e\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " Nov 02 14:42:49.281181 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281126 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-workload-socket\") pod \"d94980c9-4b40-448d-9e6d-1a8176035b8e\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " Nov 02 14:42:49.281181 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281175 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-data\") pod \"d94980c9-4b40-448d-9e6d-1a8176035b8e\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " Nov 02 14:42:49.281448 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281207 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d94980c9-4b40-448d-9e6d-1a8176035b8e-kube-api-access-5fpk6\" (UniqueName: \"kubernetes.io/projected/d94980c9-4b40-448d-9e6d-1a8176035b8e-kube-api-access-5fpk6\") pod \"d94980c9-4b40-448d-9e6d-1a8176035b8e\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " Nov 02 14:42:49.281448 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281246 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-podinfo\") pod \"d94980c9-4b40-448d-9e6d-1a8176035b8e\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " Nov 02 14:42:49.281448 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281289 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d94980c9-4b40-448d-9e6d-1a8176035b8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94980c9-4b40-448d-9e6d-1a8176035b8e-istiod-ca-cert\") pod \"d94980c9-4b40-448d-9e6d-1a8176035b8e\" (UID: \"d94980c9-4b40-448d-9e6d-1a8176035b8e\") " Nov 02 14:42:49.281448 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281336 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-workload-certs" pod "d94980c9-4b40-448d-9e6d-1a8176035b8e" (UID: "d94980c9-4b40-448d-9e6d-1a8176035b8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.281646 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281497 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-data" pod "d94980c9-4b40-448d-9e6d-1a8176035b8e" (UID: "d94980c9-4b40-448d-9e6d-1a8176035b8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.281646 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281511 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-credential-socket" pod "d94980c9-4b40-448d-9e6d-1a8176035b8e" (UID: "d94980c9-4b40-448d-9e6d-1a8176035b8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.281646 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281530 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-workload-socket" pod "d94980c9-4b40-448d-9e6d-1a8176035b8e" (UID: "d94980c9-4b40-448d-9e6d-1a8176035b8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.281801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281719 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.281801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281739 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.281801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281754 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzf5p\" (UniqueName: \"kubernetes.io/projected/0a5028a6-bf31-4b46-8252-caf390803e5e-kube-api-access-jzf5p\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.281801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281771 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.281801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281785 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.281801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281798 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.282065 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281811 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.282065 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281824 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.282065 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281838 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.282065 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281841 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d94980c9-4b40-448d-9e6d-1a8176035b8e-istiod-ca-cert" pod "d94980c9-4b40-448d-9e6d-1a8176035b8e" (UID: "d94980c9-4b40-448d-9e6d-1a8176035b8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:42:49.282065 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281850 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5028a6-bf31-4b46-8252-caf390803e5e-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.282065 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281887 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.282065 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.281906 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5028a6-bf31-4b46-8252-caf390803e5e-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.283416 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.283383 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-envoy" pod "d94980c9-4b40-448d-9e6d-1a8176035b8e" (UID: "d94980c9-4b40-448d-9e6d-1a8176035b8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.283542 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.283487 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-token" pod "d94980c9-4b40-448d-9e6d-1a8176035b8e" (UID: "d94980c9-4b40-448d-9e6d-1a8176035b8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:49.283777 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.283762 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-podinfo" pod "d94980c9-4b40-448d-9e6d-1a8176035b8e" (UID: "d94980c9-4b40-448d-9e6d-1a8176035b8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:42:49.283814 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.283783 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d94980c9-4b40-448d-9e6d-1a8176035b8e-kube-api-access-5fpk6" pod "d94980c9-4b40-448d-9e6d-1a8176035b8e" (UID: "d94980c9-4b40-448d-9e6d-1a8176035b8e"). InnerVolumeSpecName "kube-api-access-5fpk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:49.382788 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.382697 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fpk6\" (UniqueName: \"kubernetes.io/projected/d94980c9-4b40-448d-9e6d-1a8176035b8e-kube-api-access-5fpk6\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.382788 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.382734 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.382788 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.382746 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94980c9-4b40-448d-9e6d-1a8176035b8e-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.382788 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.382754 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.382788 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.382762 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d94980c9-4b40-448d-9e6d-1a8176035b8e-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.733359 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.733323 4866 generic.go:356] "Generic (PLEG): container finished" podID="d94980c9-4b40-448d-9e6d-1a8176035b8e" containerID="80d396d4def9423894c9004e8368850ac3c97cfce9ed341945f07513243dd985" exitCode=0 Nov 02 14:42:49.733606 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.733382 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" Nov 02 14:42:49.733606 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.733417 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" event={"ID":"d94980c9-4b40-448d-9e6d-1a8176035b8e","Type":"ContainerDied","Data":"80d396d4def9423894c9004e8368850ac3c97cfce9ed341945f07513243dd985"} Nov 02 14:42:49.733606 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.733473 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr" event={"ID":"d94980c9-4b40-448d-9e6d-1a8176035b8e","Type":"ContainerDied","Data":"f234e01a824cfb21532b99eab2fabac097282d01ecdf5c45454930a9ea99546a"} Nov 02 14:42:49.733606 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.733504 4866 scope.go:122] "RemoveContainer" containerID="80d396d4def9423894c9004e8368850ac3c97cfce9ed341945f07513243dd985" Nov 02 14:42:49.734963 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.734921 4866 generic.go:356] "Generic (PLEG): container finished" podID="0a5028a6-bf31-4b46-8252-caf390803e5e" containerID="15f7c0a04889c42a4ad240c2072e61bfbc58cb3e6e6fa03e9b19360468681368" exitCode=0 Nov 02 14:42:49.735144 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.734976 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" event={"ID":"0a5028a6-bf31-4b46-8252-caf390803e5e","Type":"ContainerDied","Data":"15f7c0a04889c42a4ad240c2072e61bfbc58cb3e6e6fa03e9b19360468681368"} Nov 02 14:42:49.735144 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.735001 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" Nov 02 14:42:49.735144 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.735044 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf" event={"ID":"0a5028a6-bf31-4b46-8252-caf390803e5e","Type":"ContainerDied","Data":"4075590fb4b9cf19fce7a1d6d902a0689eaae163d7a5c74b1317f603e84661f5"} Nov 02 14:42:49.743337 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.743254 4866 scope.go:122] "RemoveContainer" containerID="80d396d4def9423894c9004e8368850ac3c97cfce9ed341945f07513243dd985" Nov 02 14:42:49.743592 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:42:49.743572 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d396d4def9423894c9004e8368850ac3c97cfce9ed341945f07513243dd985\": container with ID starting with 80d396d4def9423894c9004e8368850ac3c97cfce9ed341945f07513243dd985 not found: ID does not exist" containerID="80d396d4def9423894c9004e8368850ac3c97cfce9ed341945f07513243dd985" Nov 02 14:42:49.743681 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.743598 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d396d4def9423894c9004e8368850ac3c97cfce9ed341945f07513243dd985"} err="failed to get container status \"80d396d4def9423894c9004e8368850ac3c97cfce9ed341945f07513243dd985\": rpc error: code = NotFound desc = could not find container \"80d396d4def9423894c9004e8368850ac3c97cfce9ed341945f07513243dd985\": container with ID starting with 80d396d4def9423894c9004e8368850ac3c97cfce9ed341945f07513243dd985 not found: ID does not exist" Nov 02 14:42:49.743681 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.743617 4866 scope.go:122] "RemoveContainer" containerID="15f7c0a04889c42a4ad240c2072e61bfbc58cb3e6e6fa03e9b19360468681368" Nov 02 14:42:49.754570 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.754515 4866 scope.go:122] "RemoveContainer" containerID="15f7c0a04889c42a4ad240c2072e61bfbc58cb3e6e6fa03e9b19360468681368" Nov 02 14:42:49.754977 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:42:49.754927 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15f7c0a04889c42a4ad240c2072e61bfbc58cb3e6e6fa03e9b19360468681368\": container with ID starting with 15f7c0a04889c42a4ad240c2072e61bfbc58cb3e6e6fa03e9b19360468681368 not found: ID does not exist" containerID="15f7c0a04889c42a4ad240c2072e61bfbc58cb3e6e6fa03e9b19360468681368" Nov 02 14:42:49.755119 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.754984 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15f7c0a04889c42a4ad240c2072e61bfbc58cb3e6e6fa03e9b19360468681368"} err="failed to get container status \"15f7c0a04889c42a4ad240c2072e61bfbc58cb3e6e6fa03e9b19360468681368\": rpc error: code = NotFound desc = could not find container \"15f7c0a04889c42a4ad240c2072e61bfbc58cb3e6e6fa03e9b19360468681368\": container with ID starting with 15f7c0a04889c42a4ad240c2072e61bfbc58cb3e6e6fa03e9b19360468681368 not found: ID does not exist" Nov 02 14:42:49.786172 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.786142 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf"] Nov 02 14:42:49.797818 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.797783 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-9rrgf"] Nov 02 14:42:49.825664 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.825633 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr"] Nov 02 14:42:49.829653 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:49.829633 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-j9vwr"] Nov 02 14:42:50.945223 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:50.945190 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a5028a6-bf31-4b46-8252-caf390803e5e" path="/var/lib/kubelet/pods/0a5028a6-bf31-4b46-8252-caf390803e5e/volumes" Nov 02 14:42:50.945647 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:42:50.945634 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d94980c9-4b40-448d-9e6d-1a8176035b8e" path="/var/lib/kubelet/pods/d94980c9-4b40-448d-9e6d-1a8176035b8e/volumes" Nov 02 14:43:01.977280 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:01.977194 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd"] Nov 02 14:43:01.977719 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:01.977464 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" podUID="e2a6adc3-f95e-42e0-99cc-344f7cfa647c" containerName="istio-proxy" containerID="cri-o://a43d2bec590a42c353db53144d928eb0fdfd40526a06acd906fc9d950871d7b5" gracePeriod=30 Nov 02 14:43:03.682173 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:03.682130 4866 patch_prober.go:29] interesting pod/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:43:03.682561 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:03.682203 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" podUID="e2a6adc3-f95e-42e0-99cc-344f7cfa647c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:43:06.010217 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:43:06.010178 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:43:07.232511 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.232485 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:43:07.363646 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.363552 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-kube-api-access-w6xvn\" (UniqueName: \"kubernetes.io/projected/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-kube-api-access-w6xvn\") pod \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " Nov 02 14:43:07.363646 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.363621 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istiod-ca-cert\") pod \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " Nov 02 14:43:07.363866 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.363653 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-workload-certs\") pod \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " Nov 02 14:43:07.363866 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.363675 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-workload-socket\") pod \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " Nov 02 14:43:07.363866 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.363725 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-token\" (UniqueName: \"kubernetes.io/projected/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-token\") pod \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " Nov 02 14:43:07.363866 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.363761 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-credential-socket\") pod \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " Nov 02 14:43:07.363866 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.363830 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-data\") pod \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " Nov 02 14:43:07.363866 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.363865 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-envoy\") pod \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " Nov 02 14:43:07.364226 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.363938 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-podinfo\") pod \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\" (UID: \"e2a6adc3-f95e-42e0-99cc-344f7cfa647c\") " Nov 02 14:43:07.364226 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.364070 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-workload-certs" pod "e2a6adc3-f95e-42e0-99cc-344f7cfa647c" (UID: "e2a6adc3-f95e-42e0-99cc-344f7cfa647c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:07.364226 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.364086 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-workload-socket" pod "e2a6adc3-f95e-42e0-99cc-344f7cfa647c" (UID: "e2a6adc3-f95e-42e0-99cc-344f7cfa647c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:07.364388 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.364244 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istiod-ca-cert" pod "e2a6adc3-f95e-42e0-99cc-344f7cfa647c" (UID: "e2a6adc3-f95e-42e0-99cc-344f7cfa647c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:43:07.364388 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.364347 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-data" pod "e2a6adc3-f95e-42e0-99cc-344f7cfa647c" (UID: "e2a6adc3-f95e-42e0-99cc-344f7cfa647c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:07.364494 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.364457 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:43:07.364494 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.364477 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:43:07.364494 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.364491 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:43:07.364631 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.364504 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:43:07.364631 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.364566 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-credential-socket" pod "e2a6adc3-f95e-42e0-99cc-344f7cfa647c" (UID: "e2a6adc3-f95e-42e0-99cc-344f7cfa647c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:07.365991 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.365971 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-kube-api-access-w6xvn" pod "e2a6adc3-f95e-42e0-99cc-344f7cfa647c" (UID: "e2a6adc3-f95e-42e0-99cc-344f7cfa647c"). InnerVolumeSpecName "kube-api-access-w6xvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:43:07.366148 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.366117 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-podinfo" pod "e2a6adc3-f95e-42e0-99cc-344f7cfa647c" (UID: "e2a6adc3-f95e-42e0-99cc-344f7cfa647c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:43:07.366321 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.366298 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-token" pod "e2a6adc3-f95e-42e0-99cc-344f7cfa647c" (UID: "e2a6adc3-f95e-42e0-99cc-344f7cfa647c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:43:07.366429 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.366414 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-envoy" pod "e2a6adc3-f95e-42e0-99cc-344f7cfa647c" (UID: "e2a6adc3-f95e-42e0-99cc-344f7cfa647c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:07.465052 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.465020 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6xvn\" (UniqueName: \"kubernetes.io/projected/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-kube-api-access-w6xvn\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:43:07.465052 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.465046 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:43:07.465052 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.465056 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:43:07.465292 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.465068 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:43:07.465292 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.465077 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a6adc3-f95e-42e0-99cc-344f7cfa647c-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:43:07.828674 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.828642 4866 generic.go:356] "Generic (PLEG): container finished" podID="e2a6adc3-f95e-42e0-99cc-344f7cfa647c" containerID="a43d2bec590a42c353db53144d928eb0fdfd40526a06acd906fc9d950871d7b5" exitCode=0 Nov 02 14:43:07.828878 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.828684 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" event={"ID":"e2a6adc3-f95e-42e0-99cc-344f7cfa647c","Type":"ContainerDied","Data":"a43d2bec590a42c353db53144d928eb0fdfd40526a06acd906fc9d950871d7b5"} Nov 02 14:43:07.828878 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.828693 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" Nov 02 14:43:07.828878 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.828728 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd" event={"ID":"e2a6adc3-f95e-42e0-99cc-344f7cfa647c","Type":"ContainerDied","Data":"801f667a60320858e82d29c68fe21b4e4c89daef024329823f4f280a76c20630"} Nov 02 14:43:07.828878 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.828750 4866 scope.go:122] "RemoveContainer" containerID="a43d2bec590a42c353db53144d928eb0fdfd40526a06acd906fc9d950871d7b5" Nov 02 14:43:07.838642 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.838623 4866 scope.go:122] "RemoveContainer" containerID="097e45e3e4827aba353e5137c0e539551967372656fc4fdd01a774da664111e7" Nov 02 14:43:07.846937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.846920 4866 scope.go:122] "RemoveContainer" containerID="a43d2bec590a42c353db53144d928eb0fdfd40526a06acd906fc9d950871d7b5" Nov 02 14:43:07.847223 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:43:07.847202 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43d2bec590a42c353db53144d928eb0fdfd40526a06acd906fc9d950871d7b5\": container with ID starting with a43d2bec590a42c353db53144d928eb0fdfd40526a06acd906fc9d950871d7b5 not found: ID does not exist" containerID="a43d2bec590a42c353db53144d928eb0fdfd40526a06acd906fc9d950871d7b5" Nov 02 14:43:07.847297 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.847237 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43d2bec590a42c353db53144d928eb0fdfd40526a06acd906fc9d950871d7b5"} err="failed to get container status \"a43d2bec590a42c353db53144d928eb0fdfd40526a06acd906fc9d950871d7b5\": rpc error: code = NotFound desc = could not find container \"a43d2bec590a42c353db53144d928eb0fdfd40526a06acd906fc9d950871d7b5\": container with ID starting with a43d2bec590a42c353db53144d928eb0fdfd40526a06acd906fc9d950871d7b5 not found: ID does not exist" Nov 02 14:43:07.847297 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.847266 4866 scope.go:122] "RemoveContainer" containerID="097e45e3e4827aba353e5137c0e539551967372656fc4fdd01a774da664111e7" Nov 02 14:43:07.847530 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:43:07.847512 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097e45e3e4827aba353e5137c0e539551967372656fc4fdd01a774da664111e7\": container with ID starting with 097e45e3e4827aba353e5137c0e539551967372656fc4fdd01a774da664111e7 not found: ID does not exist" containerID="097e45e3e4827aba353e5137c0e539551967372656fc4fdd01a774da664111e7" Nov 02 14:43:07.847572 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.847537 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097e45e3e4827aba353e5137c0e539551967372656fc4fdd01a774da664111e7"} err="failed to get container status \"097e45e3e4827aba353e5137c0e539551967372656fc4fdd01a774da664111e7\": rpc error: code = NotFound desc = could not find container \"097e45e3e4827aba353e5137c0e539551967372656fc4fdd01a774da664111e7\": container with ID starting with 097e45e3e4827aba353e5137c0e539551967372656fc4fdd01a774da664111e7 not found: ID does not exist" Nov 02 14:43:07.913298 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.913262 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd"] Nov 02 14:43:07.918255 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:07.918228 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-bf0b0d-openshift-default-697867ff97-84mxd"] Nov 02 14:43:08.944905 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:08.944871 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a6adc3-f95e-42e0-99cc-344f7cfa647c" path="/var/lib/kubelet/pods/e2a6adc3-f95e-42e0-99cc-344f7cfa647c/volumes" Nov 02 14:43:29.522237 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:29.522208 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64c547bcd4-s7w4p_a9a48536-da8d-42b1-ac12-d5addcc2cadc/authorino/0.log" Nov 02 14:43:32.246457 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:32.246417 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6cc7b895d5-5ldn8_4865da86-417f-4703-bcdc-c68b5de3ad59/manager/0.log" Nov 02 14:43:34.249853 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:43:34.249820 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-c2cbh_36201127-d7ae-4a96-879f-d28da880063c/proxy/0.log" Nov 02 14:43:36.009002 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:43:36.008963 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:44:06.009772 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:44:06.009737 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:44:36.010064 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:44:36.009966 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:45:06.009488 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:45:06.009440 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:45:36.009729 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:45:36.009686 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:46:06.008977 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:46:06.008873 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:46:36.009794 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:46:36.009758 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:47:06.009273 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:47:06.009234 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:47:15.603191 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:47:15.603161 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:47:15.623740 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:47:15.623713 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:47:36.009417 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:47:36.009335 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:48:06.010038 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:48:06.009999 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:48:36.009428 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:48:36.009388 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:49:06.016991 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:49:06.016877 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:49:34.665061 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.665011 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp"] Nov 02 14:49:34.665657 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.665621 4866 cpu_manager.go:414] "containerMap: removing container" podUID="e2a6adc3-f95e-42e0-99cc-344f7cfa647c" containerName="istio-proxy" Nov 02 14:49:34.665657 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.665640 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a6adc3-f95e-42e0-99cc-344f7cfa647c" containerName="istio-proxy" Nov 02 14:49:34.665761 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.665672 4866 cpu_manager.go:414] "containerMap: removing container" podUID="0a5028a6-bf31-4b46-8252-caf390803e5e" containerName="istio-proxy" Nov 02 14:49:34.665761 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.665681 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5028a6-bf31-4b46-8252-caf390803e5e" containerName="istio-proxy" Nov 02 14:49:34.665761 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.665692 4866 cpu_manager.go:414] "containerMap: removing container" podUID="d94980c9-4b40-448d-9e6d-1a8176035b8e" containerName="istio-proxy" Nov 02 14:49:34.665761 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.665696 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d94980c9-4b40-448d-9e6d-1a8176035b8e" containerName="istio-proxy" Nov 02 14:49:34.665761 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.665704 4866 cpu_manager.go:414] "containerMap: removing container" podUID="e2a6adc3-f95e-42e0-99cc-344f7cfa647c" containerName="istio-proxy" Nov 02 14:49:34.665761 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.665709 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a6adc3-f95e-42e0-99cc-344f7cfa647c" containerName="istio-proxy" Nov 02 14:49:34.665937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.665787 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5028a6-bf31-4b46-8252-caf390803e5e" containerName="istio-proxy" Nov 02 14:49:34.665937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.665796 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="d94980c9-4b40-448d-9e6d-1a8176035b8e" containerName="istio-proxy" Nov 02 14:49:34.665937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.665804 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a6adc3-f95e-42e0-99cc-344f7cfa647c" containerName="istio-proxy" Nov 02 14:49:34.668880 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.668860 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" Nov 02 14:49:34.700767 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.700734 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-87f75f6f-serving-cert\"" Nov 02 14:49:34.703844 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.703814 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp"] Nov 02 14:49:34.766806 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.766763 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74x4g\" (UniqueName: \"kubernetes.io/projected/6da817d2-26b8-4652-8c9e-3931c203e5ca-kube-api-access-74x4g\") pod \"maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp\" (UID: \"6da817d2-26b8-4652-8c9e-3931c203e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" Nov 02 14:49:34.767012 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.766812 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6da817d2-26b8-4652-8c9e-3931c203e5ca-maas-api-tls\") pod \"maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp\" (UID: \"6da817d2-26b8-4652-8c9e-3931c203e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" Nov 02 14:49:34.767086 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.767015 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da817d2-26b8-4652-8c9e-3931c203e5ca-openshift-service-ca\") pod \"maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp\" (UID: \"6da817d2-26b8-4652-8c9e-3931c203e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" Nov 02 14:49:34.767086 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.767061 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6da817d2-26b8-4652-8c9e-3931c203e5ca-metrics-certs\") pod \"maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp\" (UID: \"6da817d2-26b8-4652-8c9e-3931c203e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" Nov 02 14:49:34.867929 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.867888 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da817d2-26b8-4652-8c9e-3931c203e5ca-openshift-service-ca\") pod \"maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp\" (UID: \"6da817d2-26b8-4652-8c9e-3931c203e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" Nov 02 14:49:34.867929 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.867936 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6da817d2-26b8-4652-8c9e-3931c203e5ca-metrics-certs\") pod \"maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp\" (UID: \"6da817d2-26b8-4652-8c9e-3931c203e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" Nov 02 14:49:34.868187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.867983 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74x4g\" (UniqueName: \"kubernetes.io/projected/6da817d2-26b8-4652-8c9e-3931c203e5ca-kube-api-access-74x4g\") pod \"maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp\" (UID: \"6da817d2-26b8-4652-8c9e-3931c203e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" Nov 02 14:49:34.868187 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.868006 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6da817d2-26b8-4652-8c9e-3931c203e5ca-maas-api-tls\") pod \"maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp\" (UID: \"6da817d2-26b8-4652-8c9e-3931c203e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" Nov 02 14:49:34.868654 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.868620 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da817d2-26b8-4652-8c9e-3931c203e5ca-openshift-service-ca\") pod \"maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp\" (UID: \"6da817d2-26b8-4652-8c9e-3931c203e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" Nov 02 14:49:34.870683 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.870658 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6da817d2-26b8-4652-8c9e-3931c203e5ca-maas-api-tls\") pod \"maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp\" (UID: \"6da817d2-26b8-4652-8c9e-3931c203e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" Nov 02 14:49:34.870797 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.870658 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6da817d2-26b8-4652-8c9e-3931c203e5ca-metrics-certs\") pod \"maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp\" (UID: \"6da817d2-26b8-4652-8c9e-3931c203e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" Nov 02 14:49:34.891804 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.891774 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74x4g\" (UniqueName: \"kubernetes.io/projected/6da817d2-26b8-4652-8c9e-3931c203e5ca-kube-api-access-74x4g\") pod \"maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp\" (UID: \"6da817d2-26b8-4652-8c9e-3931c203e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" Nov 02 14:49:34.944811 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.944782 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7"] Nov 02 14:49:34.945271 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.945254 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a6adc3-f95e-42e0-99cc-344f7cfa647c" containerName="istio-proxy" Nov 02 14:49:34.948313 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.948296 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7" Nov 02 14:49:34.967976 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.967916 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7"] Nov 02 14:49:34.968339 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.968301 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-87f75f6f\"" Nov 02 14:49:34.968339 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.968311 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-87f75f6f-dockercfg-949tx\"" Nov 02 14:49:34.968601 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.968574 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c544559c-907f-403d-a931-f12697f2867f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7\" (UID: \"c544559c-907f-403d-a931-f12697f2867f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7" Nov 02 14:49:34.968698 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.968682 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z7m2\" (UniqueName: \"kubernetes.io/projected/c544559c-907f-403d-a931-f12697f2867f-kube-api-access-5z7m2\") pod \"payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7\" (UID: \"c544559c-907f-403d-a931-f12697f2867f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7" Nov 02 14:49:34.982172 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:34.982145 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" Nov 02 14:49:35.044628 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.040233 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk"] Nov 02 14:49:35.045645 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.045623 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk" Nov 02 14:49:35.076392 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.074248 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk"] Nov 02 14:49:35.076392 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.075490 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z7m2\" (UniqueName: \"kubernetes.io/projected/c544559c-907f-403d-a931-f12697f2867f-kube-api-access-5z7m2\") pod \"payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7\" (UID: \"c544559c-907f-403d-a931-f12697f2867f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7" Nov 02 14:49:35.076392 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.075566 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8799e18-932a-4aa9-b9c4-4d06e7c8beda-plugins-config-volume\") pod \"payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk\" (UID: \"c8799e18-932a-4aa9-b9c4-4d06e7c8beda\") " pod="openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk" Nov 02 14:49:35.076392 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.075658 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c544559c-907f-403d-a931-f12697f2867f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7\" (UID: \"c544559c-907f-403d-a931-f12697f2867f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7" Nov 02 14:49:35.076392 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.075695 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fcch\" (UniqueName: \"kubernetes.io/projected/c8799e18-932a-4aa9-b9c4-4d06e7c8beda-kube-api-access-6fcch\") pod \"payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk\" (UID: \"c8799e18-932a-4aa9-b9c4-4d06e7c8beda\") " pod="openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk" Nov 02 14:49:35.077121 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.077092 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c544559c-907f-403d-a931-f12697f2867f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7\" (UID: \"c544559c-907f-403d-a931-f12697f2867f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7" Nov 02 14:49:35.107911 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.107872 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z7m2\" (UniqueName: \"kubernetes.io/projected/c544559c-907f-403d-a931-f12697f2867f-kube-api-access-5z7m2\") pod \"payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7\" (UID: \"c544559c-907f-403d-a931-f12697f2867f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7" Nov 02 14:49:35.176993 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.176935 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8799e18-932a-4aa9-b9c4-4d06e7c8beda-plugins-config-volume\") pod \"payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk\" (UID: \"c8799e18-932a-4aa9-b9c4-4d06e7c8beda\") " pod="openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk" Nov 02 14:49:35.177188 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.177047 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fcch\" (UniqueName: \"kubernetes.io/projected/c8799e18-932a-4aa9-b9c4-4d06e7c8beda-kube-api-access-6fcch\") pod \"payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk\" (UID: \"c8799e18-932a-4aa9-b9c4-4d06e7c8beda\") " pod="openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk" Nov 02 14:49:35.177730 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.177708 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8799e18-932a-4aa9-b9c4-4d06e7c8beda-plugins-config-volume\") pod \"payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk\" (UID: \"c8799e18-932a-4aa9-b9c4-4d06e7c8beda\") " pod="openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk" Nov 02 14:49:35.200742 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.200664 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fcch\" (UniqueName: \"kubernetes.io/projected/c8799e18-932a-4aa9-b9c4-4d06e7c8beda-kube-api-access-6fcch\") pod \"payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk\" (UID: \"c8799e18-932a-4aa9-b9c4-4d06e7c8beda\") " pod="openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk" Nov 02 14:49:35.261306 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.261268 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7" Nov 02 14:49:35.305094 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.305066 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp"] Nov 02 14:49:35.308657 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:49:35.308629 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da817d2_26b8_4652_8c9e_3931c203e5ca.slice/crio-c69bc4a0d06ad956de1d65adad585ba0b4c4f38a7d0984fae3eb3e13833ce85f WatchSource:0}: Error finding container c69bc4a0d06ad956de1d65adad585ba0b4c4f38a7d0984fae3eb3e13833ce85f: Status 404 returned error can't find the container with id c69bc4a0d06ad956de1d65adad585ba0b4c4f38a7d0984fae3eb3e13833ce85f Nov 02 14:49:35.310604 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.310581 4866 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:49:35.380352 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.380314 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk" Nov 02 14:49:35.561156 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.561124 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7"] Nov 02 14:49:35.563789 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:49:35.563762 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc544559c_907f_403d_a931_f12697f2867f.slice/crio-85047de5d64d3836f1a96fc8cb00db5c55bab0a3c60217025f8899b0a2981d60 WatchSource:0}: Error finding container 85047de5d64d3836f1a96fc8cb00db5c55bab0a3c60217025f8899b0a2981d60: Status 404 returned error can't find the container with id 85047de5d64d3836f1a96fc8cb00db5c55bab0a3c60217025f8899b0a2981d60 Nov 02 14:49:35.681411 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.681382 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk"] Nov 02 14:49:35.684009 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:49:35.683979 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8799e18_932a_4aa9_b9c4_4d06e7c8beda.slice/crio-99346d55fd90c7360dcffab8caaaac8685f5abe3ae06424a94167992e224dd78 WatchSource:0}: Error finding container 99346d55fd90c7360dcffab8caaaac8685f5abe3ae06424a94167992e224dd78: Status 404 returned error can't find the container with id 99346d55fd90c7360dcffab8caaaac8685f5abe3ae06424a94167992e224dd78 Nov 02 14:49:35.842154 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.842054 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" event={"ID":"6da817d2-26b8-4652-8c9e-3931c203e5ca","Type":"ContainerStarted","Data":"c69bc4a0d06ad956de1d65adad585ba0b4c4f38a7d0984fae3eb3e13833ce85f"} Nov 02 14:49:35.843714 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.843679 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk" event={"ID":"c8799e18-932a-4aa9-b9c4-4d06e7c8beda","Type":"ContainerStarted","Data":"0c4e36b4fc123c5ce70afbb2e0d28c7f792ae858c9aa9c6652d9ad347df88ed5"} Nov 02 14:49:35.843860 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.843736 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk" event={"ID":"c8799e18-932a-4aa9-b9c4-4d06e7c8beda","Type":"ContainerStarted","Data":"99346d55fd90c7360dcffab8caaaac8685f5abe3ae06424a94167992e224dd78"} Nov 02 14:49:35.843929 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.843892 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk" Nov 02 14:49:35.845195 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.845165 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7" event={"ID":"c544559c-907f-403d-a931-f12697f2867f","Type":"ContainerStarted","Data":"42219c7c92a240d4dc4f2a31871e7ded40f9acf32e2ed7654bccf39665049148"} Nov 02 14:49:35.845306 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.845201 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7" event={"ID":"c544559c-907f-403d-a931-f12697f2867f","Type":"ContainerStarted","Data":"85047de5d64d3836f1a96fc8cb00db5c55bab0a3c60217025f8899b0a2981d60"} Nov 02 14:49:35.845306 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.845233 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7" Nov 02 14:49:35.885037 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.884971 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk" podStartSLOduration=0.884940277 podStartE2EDuration="884.940277ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:49:35.88397238 +0000 UTC m=+3141.067819740" watchObservedRunningTime="2026-11-02 14:49:35.884940277 +0000 UTC m=+3141.068787615" Nov 02 14:49:35.924906 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:35.924849 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7" podStartSLOduration=1.924835237 podStartE2EDuration="1.924835237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:49:35.923930705 +0000 UTC m=+3141.107778042" watchObservedRunningTime="2026-11-02 14:49:35.924835237 +0000 UTC m=+3141.108682575" Nov 02 14:49:36.010853 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:49:36.010813 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:49:36.851259 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:36.851221 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" event={"ID":"6da817d2-26b8-4652-8c9e-3931c203e5ca","Type":"ContainerStarted","Data":"91d7f0fb02e6f6f6ee7f248e7656399fe399762e652028cbb34565755d8e9514"} Nov 02 14:49:36.851691 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:36.851595 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" Nov 02 14:49:36.910356 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:36.910300 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" podStartSLOduration=2.1468975 podStartE2EDuration="2.910285762s" totalImagesPullingTime="763.388262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:49:36.908624231 +0000 UTC m=+3142.092471588" watchObservedRunningTime="2026-11-02 14:49:36.910285762 +0000 UTC m=+3142.094133137" Nov 02 14:49:42.861412 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:42.861380 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" Nov 02 14:49:46.854403 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:46.854372 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7" Nov 02 14:49:46.854937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:46.854444 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk" Nov 02 14:49:59.287600 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:59.287559 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk"] Nov 02 14:49:59.330926 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:49:59.330891 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7"] Nov 02 14:50:00.715940 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:00.715901 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp"] Nov 02 14:50:00.716390 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:00.716126 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" podUID="6da817d2-26b8-4652-8c9e-3931c203e5ca" containerName="maas-api" containerID="cri-o://91d7f0fb02e6f6f6ee7f248e7656399fe399762e652028cbb34565755d8e9514" gracePeriod=30 Nov 02 14:50:00.976192 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:00.976121 4866 generic.go:356] "Generic (PLEG): container finished" podID="6da817d2-26b8-4652-8c9e-3931c203e5ca" containerID="91d7f0fb02e6f6f6ee7f248e7656399fe399762e652028cbb34565755d8e9514" exitCode=0 Nov 02 14:50:00.976192 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:00.976173 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" event={"ID":"6da817d2-26b8-4652-8c9e-3931c203e5ca","Type":"ContainerDied","Data":"91d7f0fb02e6f6f6ee7f248e7656399fe399762e652028cbb34565755d8e9514"} Nov 02 14:50:00.976376 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:00.976213 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" event={"ID":"6da817d2-26b8-4652-8c9e-3931c203e5ca","Type":"ContainerDied","Data":"c69bc4a0d06ad956de1d65adad585ba0b4c4f38a7d0984fae3eb3e13833ce85f"} Nov 02 14:50:00.976376 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:00.976231 4866 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c69bc4a0d06ad956de1d65adad585ba0b4c4f38a7d0984fae3eb3e13833ce85f" Nov 02 14:50:00.977616 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:00.977593 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" Nov 02 14:50:01.139403 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:01.139369 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6da817d2-26b8-4652-8c9e-3931c203e5ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6da817d2-26b8-4652-8c9e-3931c203e5ca-maas-api-tls\") pod \"6da817d2-26b8-4652-8c9e-3931c203e5ca\" (UID: \"6da817d2-26b8-4652-8c9e-3931c203e5ca\") " Nov 02 14:50:01.139600 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:01.139454 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da817d2-26b8-4652-8c9e-3931c203e5ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da817d2-26b8-4652-8c9e-3931c203e5ca-openshift-service-ca\") pod \"6da817d2-26b8-4652-8c9e-3931c203e5ca\" (UID: \"6da817d2-26b8-4652-8c9e-3931c203e5ca\") " Nov 02 14:50:01.139600 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:01.139572 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da817d2-26b8-4652-8c9e-3931c203e5ca-kube-api-access-74x4g\" (UniqueName: \"kubernetes.io/projected/6da817d2-26b8-4652-8c9e-3931c203e5ca-kube-api-access-74x4g\") pod \"6da817d2-26b8-4652-8c9e-3931c203e5ca\" (UID: \"6da817d2-26b8-4652-8c9e-3931c203e5ca\") " Nov 02 14:50:01.139783 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:01.139725 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6da817d2-26b8-4652-8c9e-3931c203e5ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6da817d2-26b8-4652-8c9e-3931c203e5ca-metrics-certs\") pod \"6da817d2-26b8-4652-8c9e-3931c203e5ca\" (UID: \"6da817d2-26b8-4652-8c9e-3931c203e5ca\") " Nov 02 14:50:01.139836 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:01.139796 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da817d2-26b8-4652-8c9e-3931c203e5ca-openshift-service-ca" pod "6da817d2-26b8-4652-8c9e-3931c203e5ca" (UID: "6da817d2-26b8-4652-8c9e-3931c203e5ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:50:01.140108 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:01.140083 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da817d2-26b8-4652-8c9e-3931c203e5ca-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:50:01.141665 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:01.141644 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da817d2-26b8-4652-8c9e-3931c203e5ca-kube-api-access-74x4g" pod "6da817d2-26b8-4652-8c9e-3931c203e5ca" (UID: "6da817d2-26b8-4652-8c9e-3931c203e5ca"). InnerVolumeSpecName "kube-api-access-74x4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:50:01.141665 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:01.141651 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6da817d2-26b8-4652-8c9e-3931c203e5ca-metrics-certs" pod "6da817d2-26b8-4652-8c9e-3931c203e5ca" (UID: "6da817d2-26b8-4652-8c9e-3931c203e5ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:50:01.141798 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:01.141673 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6da817d2-26b8-4652-8c9e-3931c203e5ca-maas-api-tls" pod "6da817d2-26b8-4652-8c9e-3931c203e5ca" (UID: "6da817d2-26b8-4652-8c9e-3931c203e5ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:50:01.240669 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:01.240535 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6da817d2-26b8-4652-8c9e-3931c203e5ca-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:50:01.240669 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:01.240569 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74x4g\" (UniqueName: \"kubernetes.io/projected/6da817d2-26b8-4652-8c9e-3931c203e5ca-kube-api-access-74x4g\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:50:01.240669 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:01.240588 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6da817d2-26b8-4652-8c9e-3931c203e5ca-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:50:01.982022 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:01.981990 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp" Nov 02 14:50:02.039340 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:02.039291 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp"] Nov 02 14:50:02.050194 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:02.050157 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87f75f6f-7ff5975fd7-wp5pp"] Nov 02 14:50:02.944722 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:02.944684 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da817d2-26b8-4652-8c9e-3931c203e5ca" path="/var/lib/kubelet/pods/6da817d2-26b8-4652-8c9e-3931c203e5ca/volumes" Nov 02 14:50:04.288579 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.288532 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk" podUID="c8799e18-932a-4aa9-b9c4-4d06e7c8beda" containerName="payload-processing" containerID="cri-o://0c4e36b4fc123c5ce70afbb2e0d28c7f792ae858c9aa9c6652d9ad347df88ed5" gracePeriod=25 Nov 02 14:50:04.332140 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.332089 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7" podUID="c544559c-907f-403d-a931-f12697f2867f" containerName="payload-pre-processing" containerID="cri-o://42219c7c92a240d4dc4f2a31871e7ded40f9acf32e2ed7654bccf39665049148" gracePeriod=25 Nov 02 14:50:04.590285 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.590257 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk" Nov 02 14:50:04.616839 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.616812 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7" Nov 02 14:50:04.670800 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.670767 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8799e18-932a-4aa9-b9c4-4d06e7c8beda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8799e18-932a-4aa9-b9c4-4d06e7c8beda-plugins-config-volume\") pod \"c8799e18-932a-4aa9-b9c4-4d06e7c8beda\" (UID: \"c8799e18-932a-4aa9-b9c4-4d06e7c8beda\") " Nov 02 14:50:04.671015 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.670828 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c544559c-907f-403d-a931-f12697f2867f-kube-api-access-5z7m2\" (UniqueName: \"kubernetes.io/projected/c544559c-907f-403d-a931-f12697f2867f-kube-api-access-5z7m2\") pod \"c544559c-907f-403d-a931-f12697f2867f\" (UID: \"c544559c-907f-403d-a931-f12697f2867f\") " Nov 02 14:50:04.671015 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.670888 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c544559c-907f-403d-a931-f12697f2867f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c544559c-907f-403d-a931-f12697f2867f-plugins-config-volume\") pod \"c544559c-907f-403d-a931-f12697f2867f\" (UID: \"c544559c-907f-403d-a931-f12697f2867f\") " Nov 02 14:50:04.671015 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.670934 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8799e18-932a-4aa9-b9c4-4d06e7c8beda-kube-api-access-6fcch\" (UniqueName: \"kubernetes.io/projected/c8799e18-932a-4aa9-b9c4-4d06e7c8beda-kube-api-access-6fcch\") pod \"c8799e18-932a-4aa9-b9c4-4d06e7c8beda\" (UID: \"c8799e18-932a-4aa9-b9c4-4d06e7c8beda\") " Nov 02 14:50:04.671290 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.671263 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8799e18-932a-4aa9-b9c4-4d06e7c8beda-plugins-config-volume" pod "c8799e18-932a-4aa9-b9c4-4d06e7c8beda" (UID: "c8799e18-932a-4aa9-b9c4-4d06e7c8beda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:50:04.671345 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.671314 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c544559c-907f-403d-a931-f12697f2867f-plugins-config-volume" pod "c544559c-907f-403d-a931-f12697f2867f" (UID: "c544559c-907f-403d-a931-f12697f2867f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:50:04.673193 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.673161 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8799e18-932a-4aa9-b9c4-4d06e7c8beda-kube-api-access-6fcch" pod "c8799e18-932a-4aa9-b9c4-4d06e7c8beda" (UID: "c8799e18-932a-4aa9-b9c4-4d06e7c8beda"). InnerVolumeSpecName "kube-api-access-6fcch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:50:04.673316 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.673202 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c544559c-907f-403d-a931-f12697f2867f-kube-api-access-5z7m2" pod "c544559c-907f-403d-a931-f12697f2867f" (UID: "c544559c-907f-403d-a931-f12697f2867f"). InnerVolumeSpecName "kube-api-access-5z7m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:50:04.772251 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.772211 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8799e18-932a-4aa9-b9c4-4d06e7c8beda-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:50:04.772251 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.772242 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z7m2\" (UniqueName: \"kubernetes.io/projected/c544559c-907f-403d-a931-f12697f2867f-kube-api-access-5z7m2\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:50:04.772251 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.772253 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c544559c-907f-403d-a931-f12697f2867f-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:50:04.772251 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.772262 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fcch\" (UniqueName: \"kubernetes.io/projected/c8799e18-932a-4aa9-b9c4-4d06e7c8beda-kube-api-access-6fcch\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:50:04.997913 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.997879 4866 generic.go:356] "Generic (PLEG): container finished" podID="c8799e18-932a-4aa9-b9c4-4d06e7c8beda" containerID="0c4e36b4fc123c5ce70afbb2e0d28c7f792ae858c9aa9c6652d9ad347df88ed5" exitCode=0 Nov 02 14:50:04.998116 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.997924 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk" Nov 02 14:50:04.998116 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.997977 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk" event={"ID":"c8799e18-932a-4aa9-b9c4-4d06e7c8beda","Type":"ContainerDied","Data":"0c4e36b4fc123c5ce70afbb2e0d28c7f792ae858c9aa9c6652d9ad347df88ed5"} Nov 02 14:50:04.998116 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.998031 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk" event={"ID":"c8799e18-932a-4aa9-b9c4-4d06e7c8beda","Type":"ContainerDied","Data":"99346d55fd90c7360dcffab8caaaac8685f5abe3ae06424a94167992e224dd78"} Nov 02 14:50:04.998116 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.998057 4866 scope.go:122] "RemoveContainer" containerID="0c4e36b4fc123c5ce70afbb2e0d28c7f792ae858c9aa9c6652d9ad347df88ed5" Nov 02 14:50:04.999518 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.999484 4866 generic.go:356] "Generic (PLEG): container finished" podID="c544559c-907f-403d-a931-f12697f2867f" containerID="42219c7c92a240d4dc4f2a31871e7ded40f9acf32e2ed7654bccf39665049148" exitCode=0 Nov 02 14:50:04.999626 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.999531 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7" Nov 02 14:50:04.999626 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.999560 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7" event={"ID":"c544559c-907f-403d-a931-f12697f2867f","Type":"ContainerDied","Data":"42219c7c92a240d4dc4f2a31871e7ded40f9acf32e2ed7654bccf39665049148"} Nov 02 14:50:04.999626 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:04.999599 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7" event={"ID":"c544559c-907f-403d-a931-f12697f2867f","Type":"ContainerDied","Data":"85047de5d64d3836f1a96fc8cb00db5c55bab0a3c60217025f8899b0a2981d60"} Nov 02 14:50:05.007126 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:05.007102 4866 scope.go:122] "RemoveContainer" containerID="0c4e36b4fc123c5ce70afbb2e0d28c7f792ae858c9aa9c6652d9ad347df88ed5" Nov 02 14:50:05.007408 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:50:05.007388 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4e36b4fc123c5ce70afbb2e0d28c7f792ae858c9aa9c6652d9ad347df88ed5\": container with ID starting with 0c4e36b4fc123c5ce70afbb2e0d28c7f792ae858c9aa9c6652d9ad347df88ed5 not found: ID does not exist" containerID="0c4e36b4fc123c5ce70afbb2e0d28c7f792ae858c9aa9c6652d9ad347df88ed5" Nov 02 14:50:05.007493 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:05.007421 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4e36b4fc123c5ce70afbb2e0d28c7f792ae858c9aa9c6652d9ad347df88ed5"} err="failed to get container status \"0c4e36b4fc123c5ce70afbb2e0d28c7f792ae858c9aa9c6652d9ad347df88ed5\": rpc error: code = NotFound desc = could not find container \"0c4e36b4fc123c5ce70afbb2e0d28c7f792ae858c9aa9c6652d9ad347df88ed5\": container with ID starting with 0c4e36b4fc123c5ce70afbb2e0d28c7f792ae858c9aa9c6652d9ad347df88ed5 not found: ID does not exist" Nov 02 14:50:05.007493 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:05.007445 4866 scope.go:122] "RemoveContainer" containerID="42219c7c92a240d4dc4f2a31871e7ded40f9acf32e2ed7654bccf39665049148" Nov 02 14:50:05.016760 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:05.016738 4866 scope.go:122] "RemoveContainer" containerID="42219c7c92a240d4dc4f2a31871e7ded40f9acf32e2ed7654bccf39665049148" Nov 02 14:50:05.017086 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:50:05.017063 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42219c7c92a240d4dc4f2a31871e7ded40f9acf32e2ed7654bccf39665049148\": container with ID starting with 42219c7c92a240d4dc4f2a31871e7ded40f9acf32e2ed7654bccf39665049148 not found: ID does not exist" containerID="42219c7c92a240d4dc4f2a31871e7ded40f9acf32e2ed7654bccf39665049148" Nov 02 14:50:05.017230 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:05.017099 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42219c7c92a240d4dc4f2a31871e7ded40f9acf32e2ed7654bccf39665049148"} err="failed to get container status \"42219c7c92a240d4dc4f2a31871e7ded40f9acf32e2ed7654bccf39665049148\": rpc error: code = NotFound desc = could not find container \"42219c7c92a240d4dc4f2a31871e7ded40f9acf32e2ed7654bccf39665049148\": container with ID starting with 42219c7c92a240d4dc4f2a31871e7ded40f9acf32e2ed7654bccf39665049148 not found: ID does not exist" Nov 02 14:50:05.049521 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:05.049490 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk"] Nov 02 14:50:05.061270 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:05.061213 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87f75f6f-67d67b585c-k86hk"] Nov 02 14:50:05.085334 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:05.085290 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7"] Nov 02 14:50:05.088571 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:05.088537 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87f75f6f-55db889895-7hpp7"] Nov 02 14:50:06.008714 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:50:06.008679 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:50:06.944938 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:06.944905 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c544559c-907f-403d-a931-f12697f2867f" path="/var/lib/kubelet/pods/c544559c-907f-403d-a931-f12697f2867f/volumes" Nov 02 14:50:06.945366 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:50:06.945350 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8799e18-932a-4aa9-b9c4-4d06e7c8beda" path="/var/lib/kubelet/pods/c8799e18-932a-4aa9-b9c4-4d06e7c8beda/volumes" Nov 02 14:50:36.009148 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:50:36.009068 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:51:06.009555 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:51:06.009514 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:51:17.157982 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.157528 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb"] Nov 02 14:51:17.158581 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.158218 4866 cpu_manager.go:414] "containerMap: removing container" podUID="6da817d2-26b8-4652-8c9e-3931c203e5ca" containerName="maas-api" Nov 02 14:51:17.158581 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.158238 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da817d2-26b8-4652-8c9e-3931c203e5ca" containerName="maas-api" Nov 02 14:51:17.158581 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.158262 4866 cpu_manager.go:414] "containerMap: removing container" podUID="c8799e18-932a-4aa9-b9c4-4d06e7c8beda" containerName="payload-processing" Nov 02 14:51:17.158581 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.158271 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8799e18-932a-4aa9-b9c4-4d06e7c8beda" containerName="payload-processing" Nov 02 14:51:17.158581 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.158294 4866 cpu_manager.go:414] "containerMap: removing container" podUID="c544559c-907f-403d-a931-f12697f2867f" containerName="payload-pre-processing" Nov 02 14:51:17.158581 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.158302 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c544559c-907f-403d-a931-f12697f2867f" containerName="payload-pre-processing" Nov 02 14:51:17.158581 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.158434 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da817d2-26b8-4652-8c9e-3931c203e5ca" containerName="maas-api" Nov 02 14:51:17.158581 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.158449 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8799e18-932a-4aa9-b9c4-4d06e7c8beda" containerName="payload-processing" Nov 02 14:51:17.158581 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.158464 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="c544559c-907f-403d-a931-f12697f2867f" containerName="payload-pre-processing" Nov 02 14:51:17.161866 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.161842 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.186124 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.186085 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb"] Nov 02 14:51:17.191421 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.191389 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-170c8925-kserve-self-signed-certs\"" Nov 02 14:51:17.263977 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.263527 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-model-cache\") pod \"e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.263977 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.263585 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515a3d3a-7645-49c6-8c9c-1b185660f03e-tls-certs\") pod \"e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.263977 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.263622 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5dz7\" (UniqueName: \"kubernetes.io/projected/515a3d3a-7645-49c6-8c9c-1b185660f03e-kube-api-access-c5dz7\") pod \"e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.263977 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.263662 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-tmp-dir\") pod \"e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.263977 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.263760 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-dshm\") pod \"e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.263977 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.263787 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-home\") pod \"e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.364586 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.364532 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-dshm\") pod \"e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.364896 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.364867 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-home\") pod \"e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.366052 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.366022 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-model-cache\") pod \"e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.366155 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.366065 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515a3d3a-7645-49c6-8c9c-1b185660f03e-tls-certs\") pod \"e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.366155 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.366092 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dz7\" (UniqueName: \"kubernetes.io/projected/515a3d3a-7645-49c6-8c9c-1b185660f03e-kube-api-access-c5dz7\") pod \"e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.366155 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.366135 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-tmp-dir\") pod \"e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.366578 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.366553 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-tmp-dir\") pod \"e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.367011 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.365845 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-home\") pod \"e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.367109 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.367032 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-model-cache\") pod \"e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.372243 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.372216 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-dshm\") pod \"e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.373498 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.373476 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515a3d3a-7645-49c6-8c9c-1b185660f03e-tls-certs\") pod \"e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.390502 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.390470 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5dz7\" (UniqueName: \"kubernetes.io/projected/515a3d3a-7645-49c6-8c9c-1b185660f03e-kube-api-access-c5dz7\") pod \"e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.491065 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.491030 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:17.880846 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:17.880821 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb"] Nov 02 14:51:17.883466 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:51:17.883432 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod515a3d3a_7645_49c6_8c9c_1b185660f03e.slice/crio-274c471d661a2d498716a9e3ccdfdfc99fa0fbfb12e745154703dabeba85d12a WatchSource:0}: Error finding container 274c471d661a2d498716a9e3ccdfdfc99fa0fbfb12e745154703dabeba85d12a: Status 404 returned error can't find the container with id 274c471d661a2d498716a9e3ccdfdfc99fa0fbfb12e745154703dabeba85d12a Nov 02 14:51:18.399490 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:18.399451 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" event={"ID":"515a3d3a-7645-49c6-8c9c-1b185660f03e","Type":"ContainerStarted","Data":"32f1dc735a075b7163a1e9334113db33155107f90bfe2f54a6a295faccc8a5bf"} Nov 02 14:51:18.399906 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:18.399503 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" event={"ID":"515a3d3a-7645-49c6-8c9c-1b185660f03e","Type":"ContainerStarted","Data":"274c471d661a2d498716a9e3ccdfdfc99fa0fbfb12e745154703dabeba85d12a"} Nov 02 14:51:18.444580 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:18.444516 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" podStartSLOduration=1.444499166 podStartE2EDuration="1.444499166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:51:18.443425918 +0000 UTC m=+3243.627273283" watchObservedRunningTime="2026-11-02 14:51:18.444499166 +0000 UTC m=+3243.628346504" Nov 02 14:51:18.491650 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:18.491617 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:27.492104 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:27.492063 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:27.505405 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:27.505369 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:28.486051 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:51:28.486020 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:51:36.009916 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:51:36.009874 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:52:06.009031 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:52:06.008921 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:52:15.633455 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:15.633424 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:52:15.655571 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:15.655539 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:52:20.059201 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:20.059153 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb"] Nov 02 14:52:35.088449 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.088402 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" podUID="515a3d3a-7645-49c6-8c9c-1b185660f03e" containerName="main" containerID="cri-o://32f1dc735a075b7163a1e9334113db33155107f90bfe2f54a6a295faccc8a5bf" gracePeriod=45 Nov 02 14:52:35.340320 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.340251 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:52:35.428394 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.428357 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-model-cache\") pod \"515a3d3a-7645-49c6-8c9c-1b185660f03e\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " Nov 02 14:52:35.428601 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.428430 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-dshm\") pod \"515a3d3a-7645-49c6-8c9c-1b185660f03e\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " Nov 02 14:52:35.428601 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.428454 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-home\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-home\") pod \"515a3d3a-7645-49c6-8c9c-1b185660f03e\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " Nov 02 14:52:35.428601 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.428517 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-tmp-dir\") pod \"515a3d3a-7645-49c6-8c9c-1b185660f03e\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " Nov 02 14:52:35.428601 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.428544 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/515a3d3a-7645-49c6-8c9c-1b185660f03e-tls-certs\" (UniqueName: \"kubernetes.io/secret/515a3d3a-7645-49c6-8c9c-1b185660f03e-tls-certs\") pod \"515a3d3a-7645-49c6-8c9c-1b185660f03e\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " Nov 02 14:52:35.428828 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.428614 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/515a3d3a-7645-49c6-8c9c-1b185660f03e-kube-api-access-c5dz7\" (UniqueName: \"kubernetes.io/projected/515a3d3a-7645-49c6-8c9c-1b185660f03e-kube-api-access-c5dz7\") pod \"515a3d3a-7645-49c6-8c9c-1b185660f03e\" (UID: \"515a3d3a-7645-49c6-8c9c-1b185660f03e\") " Nov 02 14:52:35.428828 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.428698 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-model-cache" pod "515a3d3a-7645-49c6-8c9c-1b185660f03e" (UID: "515a3d3a-7645-49c6-8c9c-1b185660f03e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:52:35.428828 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.428785 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-home" pod "515a3d3a-7645-49c6-8c9c-1b185660f03e" (UID: "515a3d3a-7645-49c6-8c9c-1b185660f03e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:52:35.428828 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.428799 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-tmp-dir" pod "515a3d3a-7645-49c6-8c9c-1b185660f03e" (UID: "515a3d3a-7645-49c6-8c9c-1b185660f03e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:52:35.429060 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.429035 4866 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-model-cache\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:52:35.429060 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.429059 4866 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-home\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:52:35.429179 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.429068 4866 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-tmp-dir\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:52:35.430715 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.430692 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/515a3d3a-7645-49c6-8c9c-1b185660f03e-tls-certs" pod "515a3d3a-7645-49c6-8c9c-1b185660f03e" (UID: "515a3d3a-7645-49c6-8c9c-1b185660f03e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:52:35.430821 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.430738 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-dshm" pod "515a3d3a-7645-49c6-8c9c-1b185660f03e" (UID: "515a3d3a-7645-49c6-8c9c-1b185660f03e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:52:35.430885 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.430861 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515a3d3a-7645-49c6-8c9c-1b185660f03e-kube-api-access-c5dz7" pod "515a3d3a-7645-49c6-8c9c-1b185660f03e" (UID: "515a3d3a-7645-49c6-8c9c-1b185660f03e"). InnerVolumeSpecName "kube-api-access-c5dz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:52:35.530458 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.530420 4866 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515a3d3a-7645-49c6-8c9c-1b185660f03e-dshm\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:52:35.530458 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.530449 4866 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515a3d3a-7645-49c6-8c9c-1b185660f03e-tls-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:52:35.530458 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.530463 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5dz7\" (UniqueName: \"kubernetes.io/projected/515a3d3a-7645-49c6-8c9c-1b185660f03e-kube-api-access-c5dz7\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:52:35.831696 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.831664 4866 generic.go:356] "Generic (PLEG): container finished" podID="515a3d3a-7645-49c6-8c9c-1b185660f03e" containerID="32f1dc735a075b7163a1e9334113db33155107f90bfe2f54a6a295faccc8a5bf" exitCode=0 Nov 02 14:52:35.831910 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.831712 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" Nov 02 14:52:35.831910 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.831758 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" event={"ID":"515a3d3a-7645-49c6-8c9c-1b185660f03e","Type":"ContainerDied","Data":"32f1dc735a075b7163a1e9334113db33155107f90bfe2f54a6a295faccc8a5bf"} Nov 02 14:52:35.831910 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.831806 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb" event={"ID":"515a3d3a-7645-49c6-8c9c-1b185660f03e","Type":"ContainerDied","Data":"274c471d661a2d498716a9e3ccdfdfc99fa0fbfb12e745154703dabeba85d12a"} Nov 02 14:52:35.831910 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.831832 4866 scope.go:122] "RemoveContainer" containerID="32f1dc735a075b7163a1e9334113db33155107f90bfe2f54a6a295faccc8a5bf" Nov 02 14:52:35.841141 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.841122 4866 scope.go:122] "RemoveContainer" containerID="32f1dc735a075b7163a1e9334113db33155107f90bfe2f54a6a295faccc8a5bf" Nov 02 14:52:35.841453 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:52:35.841432 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f1dc735a075b7163a1e9334113db33155107f90bfe2f54a6a295faccc8a5bf\": container with ID starting with 32f1dc735a075b7163a1e9334113db33155107f90bfe2f54a6a295faccc8a5bf not found: ID does not exist" containerID="32f1dc735a075b7163a1e9334113db33155107f90bfe2f54a6a295faccc8a5bf" Nov 02 14:52:35.841501 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.841465 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f1dc735a075b7163a1e9334113db33155107f90bfe2f54a6a295faccc8a5bf"} err="failed to get container status \"32f1dc735a075b7163a1e9334113db33155107f90bfe2f54a6a295faccc8a5bf\": rpc error: code = NotFound desc = could not find container \"32f1dc735a075b7163a1e9334113db33155107f90bfe2f54a6a295faccc8a5bf\": container with ID starting with 32f1dc735a075b7163a1e9334113db33155107f90bfe2f54a6a295faccc8a5bf not found: ID does not exist" Nov 02 14:52:35.883017 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.882981 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb"] Nov 02 14:52:35.894434 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:35.894394 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-170c8925-kserve-7959dfcd4-2dmcb"] Nov 02 14:52:36.010844 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:52:36.010802 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:52:36.945360 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:36.945323 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="515a3d3a-7645-49c6-8c9c-1b185660f03e" path="/var/lib/kubelet/pods/515a3d3a-7645-49c6-8c9c-1b185660f03e/volumes" Nov 02 14:52:40.484780 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.484738 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z"] Nov 02 14:52:40.485352 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.485184 4866 cpu_manager.go:414] "containerMap: removing container" podUID="515a3d3a-7645-49c6-8c9c-1b185660f03e" containerName="main" Nov 02 14:52:40.485352 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.485198 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="515a3d3a-7645-49c6-8c9c-1b185660f03e" containerName="main" Nov 02 14:52:40.485352 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.485264 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="515a3d3a-7645-49c6-8c9c-1b185660f03e" containerName="main" Nov 02 14:52:40.489657 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.489634 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:40.518468 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.518433 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-2e2eed0e-kserve-self-signed-certs\"" Nov 02 14:52:40.520326 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.520303 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z"] Nov 02 14:52:40.579179 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.579128 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-model-cache\") pod \"e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:40.579179 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.579180 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-tmp-dir\") pod \"e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:40.579421 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.579289 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-dshm\") pod \"e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:40.579421 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.579316 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-tls-certs\") pod \"e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:40.579421 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.579354 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn2hx\" (UniqueName: \"kubernetes.io/projected/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-kube-api-access-mn2hx\") pod \"e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:40.579421 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.579409 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-home\") pod \"e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:40.680459 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.680408 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn2hx\" (UniqueName: \"kubernetes.io/projected/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-kube-api-access-mn2hx\") pod \"e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:40.680459 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.680467 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-home\") pod \"e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:40.680746 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.680556 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-model-cache\") pod \"e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:40.680746 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.680573 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-tmp-dir\") pod \"e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:40.680746 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.680605 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-dshm\") pod \"e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:40.680746 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.680630 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-tls-certs\") pod \"e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:40.681033 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.680998 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-home\") pod \"e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:40.681033 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.681026 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-tmp-dir\") pod \"e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:40.681181 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.681042 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-model-cache\") pod \"e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:40.683078 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.683054 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-dshm\") pod \"e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:40.683237 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.683222 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-tls-certs\") pod \"e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:40.705303 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.705263 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn2hx\" (UniqueName: \"kubernetes.io/projected/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-kube-api-access-mn2hx\") pod \"e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:40.802137 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:40.802041 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:41.100907 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:41.100675 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z"] Nov 02 14:52:41.102902 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:52:41.102877 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bf4d10e_ef7a_47c7_bc8f_202f37757cbd.slice/crio-9924a48c9d74a03adb842853aaf9f2d2b3045bd5d715917ceb651b4fc53ecad6 WatchSource:0}: Error finding container 9924a48c9d74a03adb842853aaf9f2d2b3045bd5d715917ceb651b4fc53ecad6: Status 404 returned error can't find the container with id 9924a48c9d74a03adb842853aaf9f2d2b3045bd5d715917ceb651b4fc53ecad6 Nov 02 14:52:41.863702 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:41.863660 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" event={"ID":"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd","Type":"ContainerStarted","Data":"a95c1c57165f00d03a8d7e80474f3df9a9624b323e85982215f620a4cf6219e8"} Nov 02 14:52:41.864145 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:41.863709 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" event={"ID":"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd","Type":"ContainerStarted","Data":"9924a48c9d74a03adb842853aaf9f2d2b3045bd5d715917ceb651b4fc53ecad6"} Nov 02 14:52:41.905825 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:41.905772 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" podStartSLOduration=1.905757425 podStartE2EDuration="1.905757425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:52:41.905034422 +0000 UTC m=+3327.088881771" watchObservedRunningTime="2026-11-02 14:52:41.905757425 +0000 UTC m=+3327.089604761" Nov 02 14:52:42.802990 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:42.802927 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:50.802913 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:50.802853 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:50.815573 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:50.815546 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:50.925678 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:50.925644 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:52:57.189326 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.189289 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq"] Nov 02 14:52:57.192853 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.192828 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.229407 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.229366 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-2e2eed0e-kserve-self-signed-certs\"" Nov 02 14:52:57.232332 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.232299 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq"] Nov 02 14:52:57.282298 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.282251 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-model-cache\") pod \"e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.282298 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.282302 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-home\") pod \"e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.282548 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.282326 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-tls-certs\") pod \"e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.282548 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.282417 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-dshm\") pod \"e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.282548 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.282445 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br792\" (UniqueName: \"kubernetes.io/projected/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-kube-api-access-br792\") pod \"e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.282548 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.282474 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-tmp-dir\") pod \"e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.383389 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.383345 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-dshm\") pod \"e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.383389 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.383392 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br792\" (UniqueName: \"kubernetes.io/projected/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-kube-api-access-br792\") pod \"e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.383624 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.383509 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-tmp-dir\") pod \"e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.383673 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.383648 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-model-cache\") pod \"e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.383716 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.383674 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-home\") pod \"e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.383716 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.383702 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-tls-certs\") pod \"e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.383932 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.383903 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-tmp-dir\") pod \"e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.384098 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.384001 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-home\") pod \"e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.384098 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.384074 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-model-cache\") pod \"e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.385631 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.385613 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-dshm\") pod \"e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.386060 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.386043 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-tls-certs\") pod \"e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.419896 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.419840 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br792\" (UniqueName: \"kubernetes.io/projected/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-kube-api-access-br792\") pod \"e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.507664 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.507569 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:52:57.808422 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.808334 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq"] Nov 02 14:52:57.811576 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:52:57.811538 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb32e49fd_1ac3_4a1b_9ac0_e0fb9f9d1c13.slice/crio-b7aca7f61d14786ac7757f53930202d68f3e1738f8773caf6ba087e9d89dc277 WatchSource:0}: Error finding container b7aca7f61d14786ac7757f53930202d68f3e1738f8773caf6ba087e9d89dc277: Status 404 returned error can't find the container with id b7aca7f61d14786ac7757f53930202d68f3e1738f8773caf6ba087e9d89dc277 Nov 02 14:52:57.950828 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.950795 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" event={"ID":"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13","Type":"ContainerStarted","Data":"7f222d6f692b7da30c80a58cfb81741e2097c65986dff5f939d3c652672abf48"} Nov 02 14:52:57.951030 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.950841 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" event={"ID":"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13","Type":"ContainerStarted","Data":"b7aca7f61d14786ac7757f53930202d68f3e1738f8773caf6ba087e9d89dc277"} Nov 02 14:52:57.993097 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:57.993048 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" podStartSLOduration=0.993034287 podStartE2EDuration="993.034287ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:52:57.992626323 +0000 UTC m=+3343.176473660" watchObservedRunningTime="2026-11-02 14:52:57.993034287 +0000 UTC m=+3343.176881652" Nov 02 14:52:58.508195 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:52:58.508156 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:53:06.010369 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:53:06.010328 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:53:07.507746 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:07.507699 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:53:07.520901 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:07.520872 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:53:08.013790 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:08.013760 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:53:36.009581 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:53:36.009475 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:53:37.649209 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:37.649167 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z"] Nov 02 14:53:40.691156 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:40.691118 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq"] Nov 02 14:53:46.423124 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:46.423074 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-9pqlk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:53:46.424011 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:46.423968 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-9pqlk" podUID="e3459850-2b2b-4dc0-b1ab-5ffbc87263a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:53:52.669294 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:52.669229 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" podUID="2bf4d10e-ef7a-47c7-bc8f-202f37757cbd" containerName="main" containerID="cri-o://a95c1c57165f00d03a8d7e80474f3df9a9624b323e85982215f620a4cf6219e8" gracePeriod=45 Nov 02 14:53:52.914287 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:52.914261 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:53:52.938176 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:52.938096 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-dshm\") pod \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " Nov 02 14:53:52.938176 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:52.938139 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-kube-api-access-mn2hx\" (UniqueName: \"kubernetes.io/projected/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-kube-api-access-mn2hx\") pod \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " Nov 02 14:53:52.938406 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:52.938233 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-tmp-dir\") pod \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " Nov 02 14:53:52.938406 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:52.938262 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-home\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-home\") pod \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " Nov 02 14:53:52.938406 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:52.938327 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-model-cache\") pod \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " Nov 02 14:53:52.938406 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:52.938380 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-tls-certs\" (UniqueName: \"kubernetes.io/secret/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-tls-certs\") pod \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\" (UID: \"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd\") " Nov 02 14:53:52.938627 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:52.938513 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-tmp-dir" pod "2bf4d10e-ef7a-47c7-bc8f-202f37757cbd" (UID: "2bf4d10e-ef7a-47c7-bc8f-202f37757cbd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:53:52.938729 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:52.938709 4866 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-tmp-dir\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:53:52.938729 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:52.938713 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-model-cache" pod "2bf4d10e-ef7a-47c7-bc8f-202f37757cbd" (UID: "2bf4d10e-ef7a-47c7-bc8f-202f37757cbd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:53:52.938854 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:52.938773 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-home" pod "2bf4d10e-ef7a-47c7-bc8f-202f37757cbd" (UID: "2bf4d10e-ef7a-47c7-bc8f-202f37757cbd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:53:52.940397 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:52.940360 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-kube-api-access-mn2hx" pod "2bf4d10e-ef7a-47c7-bc8f-202f37757cbd" (UID: "2bf4d10e-ef7a-47c7-bc8f-202f37757cbd"). InnerVolumeSpecName "kube-api-access-mn2hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:53:52.940502 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:52.940436 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-tls-certs" pod "2bf4d10e-ef7a-47c7-bc8f-202f37757cbd" (UID: "2bf4d10e-ef7a-47c7-bc8f-202f37757cbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:53:52.940502 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:52.940441 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-dshm" pod "2bf4d10e-ef7a-47c7-bc8f-202f37757cbd" (UID: "2bf4d10e-ef7a-47c7-bc8f-202f37757cbd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:53:53.039688 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:53.039652 4866 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-home\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:53:53.039688 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:53.039683 4866 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-model-cache\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:53:53.039688 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:53.039694 4866 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-tls-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:53:53.039688 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:53.039705 4866 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-dshm\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:53:53.039987 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:53.039720 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn2hx\" (UniqueName: \"kubernetes.io/projected/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd-kube-api-access-mn2hx\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:53:53.263839 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:53.263801 4866 generic.go:356] "Generic (PLEG): container finished" podID="2bf4d10e-ef7a-47c7-bc8f-202f37757cbd" containerID="a95c1c57165f00d03a8d7e80474f3df9a9624b323e85982215f620a4cf6219e8" exitCode=0 Nov 02 14:53:53.264074 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:53.263852 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" Nov 02 14:53:53.264074 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:53.263894 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" event={"ID":"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd","Type":"ContainerDied","Data":"a95c1c57165f00d03a8d7e80474f3df9a9624b323e85982215f620a4cf6219e8"} Nov 02 14:53:53.264074 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:53.263942 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z" event={"ID":"2bf4d10e-ef7a-47c7-bc8f-202f37757cbd","Type":"ContainerDied","Data":"9924a48c9d74a03adb842853aaf9f2d2b3045bd5d715917ceb651b4fc53ecad6"} Nov 02 14:53:53.264074 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:53.263986 4866 scope.go:122] "RemoveContainer" containerID="a95c1c57165f00d03a8d7e80474f3df9a9624b323e85982215f620a4cf6219e8" Nov 02 14:53:53.273245 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:53.273225 4866 scope.go:122] "RemoveContainer" containerID="a95c1c57165f00d03a8d7e80474f3df9a9624b323e85982215f620a4cf6219e8" Nov 02 14:53:53.273553 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:53:53.273533 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a95c1c57165f00d03a8d7e80474f3df9a9624b323e85982215f620a4cf6219e8\": container with ID starting with a95c1c57165f00d03a8d7e80474f3df9a9624b323e85982215f620a4cf6219e8 not found: ID does not exist" containerID="a95c1c57165f00d03a8d7e80474f3df9a9624b323e85982215f620a4cf6219e8" Nov 02 14:53:53.273601 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:53.273563 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95c1c57165f00d03a8d7e80474f3df9a9624b323e85982215f620a4cf6219e8"} err="failed to get container status \"a95c1c57165f00d03a8d7e80474f3df9a9624b323e85982215f620a4cf6219e8\": rpc error: code = NotFound desc = could not find container \"a95c1c57165f00d03a8d7e80474f3df9a9624b323e85982215f620a4cf6219e8\": container with ID starting with a95c1c57165f00d03a8d7e80474f3df9a9624b323e85982215f620a4cf6219e8 not found: ID does not exist" Nov 02 14:53:53.319231 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:53.319196 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z"] Nov 02 14:53:53.330838 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:53.330801 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-2e2eed0e-kserve-59d59f66bf-h254z"] Nov 02 14:53:54.945826 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:54.945792 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf4d10e-ef7a-47c7-bc8f-202f37757cbd" path="/var/lib/kubelet/pods/2bf4d10e-ef7a-47c7-bc8f-202f37757cbd/volumes" Nov 02 14:53:55.720302 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:55.720262 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" podUID="b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13" containerName="main" containerID="cri-o://7f222d6f692b7da30c80a58cfb81741e2097c65986dff5f939d3c652672abf48" gracePeriod=45 Nov 02 14:53:55.980293 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:55.980232 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:53:56.070033 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.070000 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-model-cache\") pod \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " Nov 02 14:53:56.070215 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.070040 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-tls-certs\" (UniqueName: \"kubernetes.io/secret/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-tls-certs\") pod \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " Nov 02 14:53:56.070215 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.070097 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-kube-api-access-br792\" (UniqueName: \"kubernetes.io/projected/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-kube-api-access-br792\") pod \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " Nov 02 14:53:56.070215 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.070133 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-dshm\") pod \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " Nov 02 14:53:56.070335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.070224 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-tmp-dir\") pod \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " Nov 02 14:53:56.070335 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.070278 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-home\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-home\") pod \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\" (UID: \"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13\") " Nov 02 14:53:56.070417 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.070334 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-model-cache" pod "b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13" (UID: "b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:53:56.070500 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.070470 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-tmp-dir" pod "b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13" (UID: "b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:53:56.070683 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.070659 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-home" pod "b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13" (UID: "b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:53:56.070756 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.070686 4866 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-tmp-dir\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:53:56.070756 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.070709 4866 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-model-cache\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:53:56.072340 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.072318 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-dshm" pod "b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13" (UID: "b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:53:56.072452 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.072317 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-tls-certs" pod "b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13" (UID: "b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:53:56.072452 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.072366 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-kube-api-access-br792" pod "b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13" (UID: "b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13"). InnerVolumeSpecName "kube-api-access-br792". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:53:56.171396 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.171362 4866 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-home\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:53:56.171396 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.171389 4866 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-tls-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:53:56.171396 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.171402 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br792\" (UniqueName: \"kubernetes.io/projected/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-kube-api-access-br792\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:53:56.171640 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.171411 4866 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13-dshm\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:53:56.280399 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.280302 4866 generic.go:356] "Generic (PLEG): container finished" podID="b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13" containerID="7f222d6f692b7da30c80a58cfb81741e2097c65986dff5f939d3c652672abf48" exitCode=0 Nov 02 14:53:56.280399 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.280345 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" event={"ID":"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13","Type":"ContainerDied","Data":"7f222d6f692b7da30c80a58cfb81741e2097c65986dff5f939d3c652672abf48"} Nov 02 14:53:56.280399 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.280360 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" Nov 02 14:53:56.280399 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.280398 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq" event={"ID":"b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13","Type":"ContainerDied","Data":"b7aca7f61d14786ac7757f53930202d68f3e1738f8773caf6ba087e9d89dc277"} Nov 02 14:53:56.280716 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.280420 4866 scope.go:122] "RemoveContainer" containerID="7f222d6f692b7da30c80a58cfb81741e2097c65986dff5f939d3c652672abf48" Nov 02 14:53:56.293865 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.293824 4866 scope.go:122] "RemoveContainer" containerID="7f222d6f692b7da30c80a58cfb81741e2097c65986dff5f939d3c652672abf48" Nov 02 14:53:56.294188 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:53:56.294164 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f222d6f692b7da30c80a58cfb81741e2097c65986dff5f939d3c652672abf48\": container with ID starting with 7f222d6f692b7da30c80a58cfb81741e2097c65986dff5f939d3c652672abf48 not found: ID does not exist" containerID="7f222d6f692b7da30c80a58cfb81741e2097c65986dff5f939d3c652672abf48" Nov 02 14:53:56.294268 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.294206 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f222d6f692b7da30c80a58cfb81741e2097c65986dff5f939d3c652672abf48"} err="failed to get container status \"7f222d6f692b7da30c80a58cfb81741e2097c65986dff5f939d3c652672abf48\": rpc error: code = NotFound desc = could not find container \"7f222d6f692b7da30c80a58cfb81741e2097c65986dff5f939d3c652672abf48\": container with ID starting with 7f222d6f692b7da30c80a58cfb81741e2097c65986dff5f939d3c652672abf48 not found: ID does not exist" Nov 02 14:53:56.349273 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.349237 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq"] Nov 02 14:53:56.360985 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.360937 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-2e2eed0e-kserve-7f9969b75c-22wpq"] Nov 02 14:53:56.986379 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:53:56.986344 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13" path="/var/lib/kubelet/pods/b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13/volumes" Nov 02 14:54:06.009892 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:54:06.009850 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:54:22.258705 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.258665 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn"] Nov 02 14:54:22.259734 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.259696 4866 cpu_manager.go:414] "containerMap: removing container" podUID="b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13" containerName="main" Nov 02 14:54:22.259734 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.259725 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13" containerName="main" Nov 02 14:54:22.259903 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.259743 4866 cpu_manager.go:414] "containerMap: removing container" podUID="2bf4d10e-ef7a-47c7-bc8f-202f37757cbd" containerName="main" Nov 02 14:54:22.259903 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.259748 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf4d10e-ef7a-47c7-bc8f-202f37757cbd" containerName="main" Nov 02 14:54:22.259903 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.259809 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="b32e49fd-1ac3-4a1b-9ac0-e0fb9f9d1c13" containerName="main" Nov 02 14:54:22.259903 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.259818 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf4d10e-ef7a-47c7-bc8f-202f37757cbd" containerName="main" Nov 02 14:54:22.264381 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.264349 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" Nov 02 14:54:22.283793 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.283761 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-b21aae-serving-cert\"" Nov 02 14:54:22.295179 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.295141 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn"] Nov 02 14:54:22.367145 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.367107 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09be7298-cfa0-4d2c-8535-40ff55ec6eba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn\" (UID: \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" Nov 02 14:54:22.367312 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.367262 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09be7298-cfa0-4d2c-8535-40ff55ec6eba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn\" (UID: \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" Nov 02 14:54:22.367379 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.367359 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09be7298-cfa0-4d2c-8535-40ff55ec6eba-metrics-certs\") pod \"maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn\" (UID: \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" Nov 02 14:54:22.367425 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.367414 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7669q\" (UniqueName: \"kubernetes.io/projected/09be7298-cfa0-4d2c-8535-40ff55ec6eba-kube-api-access-7669q\") pod \"maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn\" (UID: \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" Nov 02 14:54:22.468764 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.468723 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09be7298-cfa0-4d2c-8535-40ff55ec6eba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn\" (UID: \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" Nov 02 14:54:22.468979 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.468785 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09be7298-cfa0-4d2c-8535-40ff55ec6eba-metrics-certs\") pod \"maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn\" (UID: \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" Nov 02 14:54:22.468979 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.468822 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7669q\" (UniqueName: \"kubernetes.io/projected/09be7298-cfa0-4d2c-8535-40ff55ec6eba-kube-api-access-7669q\") pod \"maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn\" (UID: \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" Nov 02 14:54:22.468979 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.468844 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09be7298-cfa0-4d2c-8535-40ff55ec6eba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn\" (UID: \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" Nov 02 14:54:22.469511 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.469486 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09be7298-cfa0-4d2c-8535-40ff55ec6eba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn\" (UID: \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" Nov 02 14:54:22.471379 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.471352 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09be7298-cfa0-4d2c-8535-40ff55ec6eba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn\" (UID: \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" Nov 02 14:54:22.471468 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.471413 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09be7298-cfa0-4d2c-8535-40ff55ec6eba-metrics-certs\") pod \"maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn\" (UID: \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" Nov 02 14:54:22.500449 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.500420 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7669q\" (UniqueName: \"kubernetes.io/projected/09be7298-cfa0-4d2c-8535-40ff55ec6eba-kube-api-access-7669q\") pod \"maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn\" (UID: \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" Nov 02 14:54:22.539048 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.538977 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95"] Nov 02 14:54:22.542343 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.542325 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95" Nov 02 14:54:22.557700 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.557675 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b21aae-dockercfg-878fc\"" Nov 02 14:54:22.557843 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.557775 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b21aae\"" Nov 02 14:54:22.569965 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.569924 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95"] Nov 02 14:54:22.578369 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.578339 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" Nov 02 14:54:22.642028 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.641621 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg"] Nov 02 14:54:22.658191 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.652262 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg" Nov 02 14:54:22.674303 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.672180 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74qz5\" (UniqueName: \"kubernetes.io/projected/4740913f-d967-4cef-a102-a7a9bac42715-kube-api-access-74qz5\") pod \"payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95\" (UID: \"4740913f-d967-4cef-a102-a7a9bac42715\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95" Nov 02 14:54:22.674303 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.672361 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4740913f-d967-4cef-a102-a7a9bac42715-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95\" (UID: \"4740913f-d967-4cef-a102-a7a9bac42715\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95" Nov 02 14:54:22.677791 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.677708 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg"] Nov 02 14:54:22.773742 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.773698 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gkmm\" (UniqueName: \"kubernetes.io/projected/43a26150-aac7-4769-8dfe-ff837566af27-kube-api-access-8gkmm\") pod \"payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg\" (UID: \"43a26150-aac7-4769-8dfe-ff837566af27\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg" Nov 02 14:54:22.773979 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.773765 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4740913f-d967-4cef-a102-a7a9bac42715-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95\" (UID: \"4740913f-d967-4cef-a102-a7a9bac42715\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95" Nov 02 14:54:22.773979 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.773865 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74qz5\" (UniqueName: \"kubernetes.io/projected/4740913f-d967-4cef-a102-a7a9bac42715-kube-api-access-74qz5\") pod \"payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95\" (UID: \"4740913f-d967-4cef-a102-a7a9bac42715\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95" Nov 02 14:54:22.773979 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.773909 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43a26150-aac7-4769-8dfe-ff837566af27-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg\" (UID: \"43a26150-aac7-4769-8dfe-ff837566af27\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg" Nov 02 14:54:22.774554 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.774533 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4740913f-d967-4cef-a102-a7a9bac42715-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95\" (UID: \"4740913f-d967-4cef-a102-a7a9bac42715\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95" Nov 02 14:54:22.797893 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.797820 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74qz5\" (UniqueName: \"kubernetes.io/projected/4740913f-d967-4cef-a102-a7a9bac42715-kube-api-access-74qz5\") pod \"payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95\" (UID: \"4740913f-d967-4cef-a102-a7a9bac42715\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95" Nov 02 14:54:22.857648 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.857608 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95" Nov 02 14:54:22.874822 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.874792 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gkmm\" (UniqueName: \"kubernetes.io/projected/43a26150-aac7-4769-8dfe-ff837566af27-kube-api-access-8gkmm\") pod \"payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg\" (UID: \"43a26150-aac7-4769-8dfe-ff837566af27\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg" Nov 02 14:54:22.875026 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.874855 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43a26150-aac7-4769-8dfe-ff837566af27-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg\" (UID: \"43a26150-aac7-4769-8dfe-ff837566af27\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg" Nov 02 14:54:22.875576 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.875551 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43a26150-aac7-4769-8dfe-ff837566af27-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg\" (UID: \"43a26150-aac7-4769-8dfe-ff837566af27\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg" Nov 02 14:54:22.899541 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.899506 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gkmm\" (UniqueName: \"kubernetes.io/projected/43a26150-aac7-4769-8dfe-ff837566af27-kube-api-access-8gkmm\") pod \"payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg\" (UID: \"43a26150-aac7-4769-8dfe-ff837566af27\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg" Nov 02 14:54:22.901326 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.901302 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn"] Nov 02 14:54:22.903782 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:54:22.903756 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09be7298_cfa0_4d2c_8535_40ff55ec6eba.slice/crio-add0b1603ead0170198b60f4f83e9abfbbf4bd37a5f3f727228e8a3486354218 WatchSource:0}: Error finding container add0b1603ead0170198b60f4f83e9abfbbf4bd37a5f3f727228e8a3486354218: Status 404 returned error can't find the container with id add0b1603ead0170198b60f4f83e9abfbbf4bd37a5f3f727228e8a3486354218 Nov 02 14:54:22.982050 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:22.982012 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg" Nov 02 14:54:23.157465 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:23.157435 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95"] Nov 02 14:54:23.159661 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:54:23.159622 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4740913f_d967_4cef_a102_a7a9bac42715.slice/crio-b07d8b19463e79fb1bb0803fbbd5ed67c8645b652ed6c424c03eb3d0a366f804 WatchSource:0}: Error finding container b07d8b19463e79fb1bb0803fbbd5ed67c8645b652ed6c424c03eb3d0a366f804: Status 404 returned error can't find the container with id b07d8b19463e79fb1bb0803fbbd5ed67c8645b652ed6c424c03eb3d0a366f804 Nov 02 14:54:23.283525 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:23.283261 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg"] Nov 02 14:54:23.285573 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:54:23.285548 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a26150_aac7_4769_8dfe_ff837566af27.slice/crio-7ad7ea5bd0c732102e1277b501a9e100a8139adfa491925693fb6c51c29c6bc4 WatchSource:0}: Error finding container 7ad7ea5bd0c732102e1277b501a9e100a8139adfa491925693fb6c51c29c6bc4: Status 404 returned error can't find the container with id 7ad7ea5bd0c732102e1277b501a9e100a8139adfa491925693fb6c51c29c6bc4 Nov 02 14:54:23.434600 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:23.434494 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" event={"ID":"09be7298-cfa0-4d2c-8535-40ff55ec6eba","Type":"ContainerStarted","Data":"07479cd58fbc8367e83081966c34ad672426acb2c8b7c055cb7151ff79ae1e49"} Nov 02 14:54:23.434600 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:23.434556 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" event={"ID":"09be7298-cfa0-4d2c-8535-40ff55ec6eba","Type":"ContainerStarted","Data":"add0b1603ead0170198b60f4f83e9abfbbf4bd37a5f3f727228e8a3486354218"} Nov 02 14:54:23.434846 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:23.434693 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" Nov 02 14:54:23.436206 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:23.436177 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg" event={"ID":"43a26150-aac7-4769-8dfe-ff837566af27","Type":"ContainerStarted","Data":"937d691a07a2d3ac47abf163d19a692a05c2a923f50a6eb864cdc495825e0eb3"} Nov 02 14:54:23.436353 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:23.436222 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg" event={"ID":"43a26150-aac7-4769-8dfe-ff837566af27","Type":"ContainerStarted","Data":"7ad7ea5bd0c732102e1277b501a9e100a8139adfa491925693fb6c51c29c6bc4"} Nov 02 14:54:23.436353 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:23.436249 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg" Nov 02 14:54:23.437818 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:23.437799 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95" event={"ID":"4740913f-d967-4cef-a102-a7a9bac42715","Type":"ContainerStarted","Data":"0f0ffb3d60a810db2799cee2c5e8e1069c9211b6e94c8ea08f17ca31e1be7fb9"} Nov 02 14:54:23.437905 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:23.437831 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95" event={"ID":"4740913f-d967-4cef-a102-a7a9bac42715","Type":"ContainerStarted","Data":"b07d8b19463e79fb1bb0803fbbd5ed67c8645b652ed6c424c03eb3d0a366f804"} Nov 02 14:54:23.437980 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:23.437914 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95" Nov 02 14:54:23.476860 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:23.476798 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" podStartSLOduration=1.085344127 podStartE2EDuration="1.476785617s" totalImagesPullingTime="391.44149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:54:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:54:23.476292141 +0000 UTC m=+3428.660139478" watchObservedRunningTime="2026-11-02 14:54:23.476785617 +0000 UTC m=+3428.660632953" Nov 02 14:54:23.520834 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:23.520763 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95" podStartSLOduration=1.520747583 podStartE2EDuration="1.520747583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:54:23.519072781 +0000 UTC m=+3428.702920118" watchObservedRunningTime="2026-11-02 14:54:23.520747583 +0000 UTC m=+3428.704594921" Nov 02 14:54:23.560215 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:23.560152 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg" podStartSLOduration=1.5601383819999999 podStartE2EDuration="1.560138382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:54:23.559670302 +0000 UTC m=+3428.743517638" watchObservedRunningTime="2026-11-02 14:54:23.560138382 +0000 UTC m=+3428.743985719" Nov 02 14:54:29.448432 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:29.448399 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" Nov 02 14:54:34.445817 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:34.445783 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg" Nov 02 14:54:34.446301 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:34.445860 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95" Nov 02 14:54:35.495087 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:35.495055 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg"] Nov 02 14:54:35.525359 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:35.525316 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95"] Nov 02 14:54:36.009549 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:54:36.009515 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:54:40.495515 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:40.495446 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg" podUID="43a26150-aac7-4769-8dfe-ff837566af27" containerName="payload-processing" containerID="cri-o://937d691a07a2d3ac47abf163d19a692a05c2a923f50a6eb864cdc495825e0eb3" gracePeriod=25 Nov 02 14:54:40.526096 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:40.526057 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95" podUID="4740913f-d967-4cef-a102-a7a9bac42715" containerName="payload-pre-processing" containerID="cri-o://0f0ffb3d60a810db2799cee2c5e8e1069c9211b6e94c8ea08f17ca31e1be7fb9" gracePeriod=25 Nov 02 14:54:40.776042 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:40.776014 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg" Nov 02 14:54:40.809711 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:40.809687 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95" Nov 02 14:54:40.870837 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:40.870809 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4740913f-d967-4cef-a102-a7a9bac42715-kube-api-access-74qz5\" (UniqueName: \"kubernetes.io/projected/4740913f-d967-4cef-a102-a7a9bac42715-kube-api-access-74qz5\") pod \"4740913f-d967-4cef-a102-a7a9bac42715\" (UID: \"4740913f-d967-4cef-a102-a7a9bac42715\") " Nov 02 14:54:40.871063 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:40.871033 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4740913f-d967-4cef-a102-a7a9bac42715-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4740913f-d967-4cef-a102-a7a9bac42715-plugins-config-volume\") pod \"4740913f-d967-4cef-a102-a7a9bac42715\" (UID: \"4740913f-d967-4cef-a102-a7a9bac42715\") " Nov 02 14:54:40.871138 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:40.871117 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a26150-aac7-4769-8dfe-ff837566af27-kube-api-access-8gkmm\" (UniqueName: \"kubernetes.io/projected/43a26150-aac7-4769-8dfe-ff837566af27-kube-api-access-8gkmm\") pod \"43a26150-aac7-4769-8dfe-ff837566af27\" (UID: \"43a26150-aac7-4769-8dfe-ff837566af27\") " Nov 02 14:54:40.871219 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:40.871200 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a26150-aac7-4769-8dfe-ff837566af27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43a26150-aac7-4769-8dfe-ff837566af27-plugins-config-volume\") pod \"43a26150-aac7-4769-8dfe-ff837566af27\" (UID: \"43a26150-aac7-4769-8dfe-ff837566af27\") " Nov 02 14:54:40.871442 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:40.871422 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4740913f-d967-4cef-a102-a7a9bac42715-plugins-config-volume" pod "4740913f-d967-4cef-a102-a7a9bac42715" (UID: "4740913f-d967-4cef-a102-a7a9bac42715"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:54:40.871753 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:40.871727 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4740913f-d967-4cef-a102-a7a9bac42715-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:54:40.871889 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:40.871751 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a26150-aac7-4769-8dfe-ff837566af27-plugins-config-volume" pod "43a26150-aac7-4769-8dfe-ff837566af27" (UID: "43a26150-aac7-4769-8dfe-ff837566af27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:54:40.873688 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:40.873658 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4740913f-d967-4cef-a102-a7a9bac42715-kube-api-access-74qz5" pod "4740913f-d967-4cef-a102-a7a9bac42715" (UID: "4740913f-d967-4cef-a102-a7a9bac42715"). InnerVolumeSpecName "kube-api-access-74qz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:54:40.873805 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:40.873759 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a26150-aac7-4769-8dfe-ff837566af27-kube-api-access-8gkmm" pod "43a26150-aac7-4769-8dfe-ff837566af27" (UID: "43a26150-aac7-4769-8dfe-ff837566af27"). InnerVolumeSpecName "kube-api-access-8gkmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:54:40.972326 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:40.972289 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43a26150-aac7-4769-8dfe-ff837566af27-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:54:40.972326 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:40.972323 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74qz5\" (UniqueName: \"kubernetes.io/projected/4740913f-d967-4cef-a102-a7a9bac42715-kube-api-access-74qz5\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:54:40.972513 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:40.972335 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gkmm\" (UniqueName: \"kubernetes.io/projected/43a26150-aac7-4769-8dfe-ff837566af27-kube-api-access-8gkmm\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:54:41.527760 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:41.527725 4866 generic.go:356] "Generic (PLEG): container finished" podID="43a26150-aac7-4769-8dfe-ff837566af27" containerID="937d691a07a2d3ac47abf163d19a692a05c2a923f50a6eb864cdc495825e0eb3" exitCode=0 Nov 02 14:54:41.528248 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:41.527791 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg" Nov 02 14:54:41.528248 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:41.527804 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg" event={"ID":"43a26150-aac7-4769-8dfe-ff837566af27","Type":"ContainerDied","Data":"937d691a07a2d3ac47abf163d19a692a05c2a923f50a6eb864cdc495825e0eb3"} Nov 02 14:54:41.528248 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:41.527892 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg" event={"ID":"43a26150-aac7-4769-8dfe-ff837566af27","Type":"ContainerDied","Data":"7ad7ea5bd0c732102e1277b501a9e100a8139adfa491925693fb6c51c29c6bc4"} Nov 02 14:54:41.528248 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:41.527922 4866 scope.go:122] "RemoveContainer" containerID="937d691a07a2d3ac47abf163d19a692a05c2a923f50a6eb864cdc495825e0eb3" Nov 02 14:54:41.529309 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:41.529128 4866 generic.go:356] "Generic (PLEG): container finished" podID="4740913f-d967-4cef-a102-a7a9bac42715" containerID="0f0ffb3d60a810db2799cee2c5e8e1069c9211b6e94c8ea08f17ca31e1be7fb9" exitCode=0 Nov 02 14:54:41.529309 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:41.529201 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95" Nov 02 14:54:41.529309 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:41.529216 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95" event={"ID":"4740913f-d967-4cef-a102-a7a9bac42715","Type":"ContainerDied","Data":"0f0ffb3d60a810db2799cee2c5e8e1069c9211b6e94c8ea08f17ca31e1be7fb9"} Nov 02 14:54:41.529309 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:41.529261 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95" event={"ID":"4740913f-d967-4cef-a102-a7a9bac42715","Type":"ContainerDied","Data":"b07d8b19463e79fb1bb0803fbbd5ed67c8645b652ed6c424c03eb3d0a366f804"} Nov 02 14:54:41.536517 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:41.536497 4866 scope.go:122] "RemoveContainer" containerID="937d691a07a2d3ac47abf163d19a692a05c2a923f50a6eb864cdc495825e0eb3" Nov 02 14:54:41.536748 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:54:41.536728 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"937d691a07a2d3ac47abf163d19a692a05c2a923f50a6eb864cdc495825e0eb3\": container with ID starting with 937d691a07a2d3ac47abf163d19a692a05c2a923f50a6eb864cdc495825e0eb3 not found: ID does not exist" containerID="937d691a07a2d3ac47abf163d19a692a05c2a923f50a6eb864cdc495825e0eb3" Nov 02 14:54:41.536801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:41.536764 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"937d691a07a2d3ac47abf163d19a692a05c2a923f50a6eb864cdc495825e0eb3"} err="failed to get container status \"937d691a07a2d3ac47abf163d19a692a05c2a923f50a6eb864cdc495825e0eb3\": rpc error: code = NotFound desc = could not find container \"937d691a07a2d3ac47abf163d19a692a05c2a923f50a6eb864cdc495825e0eb3\": container with ID starting with 937d691a07a2d3ac47abf163d19a692a05c2a923f50a6eb864cdc495825e0eb3 not found: ID does not exist" Nov 02 14:54:41.536801 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:41.536783 4866 scope.go:122] "RemoveContainer" containerID="0f0ffb3d60a810db2799cee2c5e8e1069c9211b6e94c8ea08f17ca31e1be7fb9" Nov 02 14:54:41.544537 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:41.544517 4866 scope.go:122] "RemoveContainer" containerID="0f0ffb3d60a810db2799cee2c5e8e1069c9211b6e94c8ea08f17ca31e1be7fb9" Nov 02 14:54:41.544816 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:54:41.544797 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f0ffb3d60a810db2799cee2c5e8e1069c9211b6e94c8ea08f17ca31e1be7fb9\": container with ID starting with 0f0ffb3d60a810db2799cee2c5e8e1069c9211b6e94c8ea08f17ca31e1be7fb9 not found: ID does not exist" containerID="0f0ffb3d60a810db2799cee2c5e8e1069c9211b6e94c8ea08f17ca31e1be7fb9" Nov 02 14:54:41.544876 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:41.544823 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f0ffb3d60a810db2799cee2c5e8e1069c9211b6e94c8ea08f17ca31e1be7fb9"} err="failed to get container status \"0f0ffb3d60a810db2799cee2c5e8e1069c9211b6e94c8ea08f17ca31e1be7fb9\": rpc error: code = NotFound desc = could not find container \"0f0ffb3d60a810db2799cee2c5e8e1069c9211b6e94c8ea08f17ca31e1be7fb9\": container with ID starting with 0f0ffb3d60a810db2799cee2c5e8e1069c9211b6e94c8ea08f17ca31e1be7fb9 not found: ID does not exist" Nov 02 14:54:41.583382 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:41.583351 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg"] Nov 02 14:54:41.593862 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:41.593830 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b21aae-748f845497-z4ztg"] Nov 02 14:54:41.630111 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:41.630074 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95"] Nov 02 14:54:41.634088 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:41.634060 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-6c6bc46cdc-f4v95"] Nov 02 14:54:42.945096 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:42.945050 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a26150-aac7-4769-8dfe-ff837566af27" path="/var/lib/kubelet/pods/43a26150-aac7-4769-8dfe-ff837566af27/volumes" Nov 02 14:54:42.945631 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:42.945608 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4740913f-d967-4cef-a102-a7a9bac42715" path="/var/lib/kubelet/pods/4740913f-d967-4cef-a102-a7a9bac42715/volumes" Nov 02 14:54:57.927753 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:57.927652 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv"] Nov 02 14:54:57.928684 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:57.928395 4866 cpu_manager.go:414] "containerMap: removing container" podUID="43a26150-aac7-4769-8dfe-ff837566af27" containerName="payload-processing" Nov 02 14:54:57.928684 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:57.928418 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a26150-aac7-4769-8dfe-ff837566af27" containerName="payload-processing" Nov 02 14:54:57.928684 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:57.928459 4866 cpu_manager.go:414] "containerMap: removing container" podUID="4740913f-d967-4cef-a102-a7a9bac42715" containerName="payload-pre-processing" Nov 02 14:54:57.928684 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:57.928466 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4740913f-d967-4cef-a102-a7a9bac42715" containerName="payload-pre-processing" Nov 02 14:54:57.928684 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:57.928581 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a26150-aac7-4769-8dfe-ff837566af27" containerName="payload-processing" Nov 02 14:54:57.928684 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:57.928598 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="4740913f-d967-4cef-a102-a7a9bac42715" containerName="payload-pre-processing" Nov 02 14:54:57.932190 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:57.932169 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:57.946778 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:57.946750 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-be6018-openshift-default-dockercfg-wkvvm\"" Nov 02 14:54:57.960388 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:57.960351 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv"] Nov 02 14:54:58.041454 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.041414 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-envoy\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.041663 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.041457 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-data\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.041663 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.041485 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-token\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.041663 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.041575 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istiod-ca-cert\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.041787 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.041723 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcbbb\" (UniqueName: \"kubernetes.io/projected/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-kube-api-access-xcbbb\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.041854 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.041836 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-workload-socket\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.041906 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.041897 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-credential-socket\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.041942 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.041927 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-workload-certs\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.041942 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.041967 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-podinfo\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.143216 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.143176 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-podinfo\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.143433 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.143246 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-envoy\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.143433 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.143286 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-data\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.143433 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.143318 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-token\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.143433 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.143347 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istiod-ca-cert\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.143433 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.143390 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcbbb\" (UniqueName: \"kubernetes.io/projected/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-kube-api-access-xcbbb\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.143716 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.143439 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-workload-socket\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.143716 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.143483 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-credential-socket\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.143716 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.143516 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-workload-certs\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.143882 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.143843 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-workload-socket\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.143937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.143892 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-credential-socket\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.144024 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.143995 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-data\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.144212 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.144190 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istiod-ca-cert\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.144280 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.144259 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-workload-certs\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.145706 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.145684 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-envoy\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.145885 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.145864 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-podinfo\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.168645 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.168614 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-token\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.168970 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.168917 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcbbb\" (UniqueName: \"kubernetes.io/projected/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-kube-api-access-xcbbb\") pod \"e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.249114 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.249079 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:54:58.562319 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.562135 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv"] Nov 02 14:54:58.565231 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:54:58.565203 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8b92c0_1cbe_4ed4_bc3f_fde5545a25f4.slice/crio-9c63c84656a55931694a56055f708d1d7c6b2532741046174bf659e5bd4cc51d WatchSource:0}: Error finding container 9c63c84656a55931694a56055f708d1d7c6b2532741046174bf659e5bd4cc51d: Status 404 returned error can't find the container with id 9c63c84656a55931694a56055f708d1d7c6b2532741046174bf659e5bd4cc51d Nov 02 14:54:58.567388 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.567356 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:54:58.567475 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.567422 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:54:58.567475 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.567461 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:54:58.609842 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:58.609816 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" event={"ID":"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4","Type":"ContainerStarted","Data":"9c63c84656a55931694a56055f708d1d7c6b2532741046174bf659e5bd4cc51d"} Nov 02 14:54:59.616716 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:54:59.616670 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" event={"ID":"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4","Type":"ContainerStarted","Data":"a49e97c5284865666d4eb221665e836a2d09a9cdc8ea21f600a8b7bb33e2837e"} Nov 02 14:55:00.250176 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:00.250138 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:55:00.255681 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:00.255650 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:55:00.305732 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:00.305677 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" podStartSLOduration=3.305662459 podStartE2EDuration="3.305662459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:54:59.684272451 +0000 UTC m=+3464.868119788" watchObservedRunningTime="2026-11-02 14:55:00.305662459 +0000 UTC m=+3465.489509797" Nov 02 14:55:00.622230 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:00.622142 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:55:00.623194 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:00.623167 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:55:04.026622 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.026589 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq"] Nov 02 14:55:04.031088 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.031058 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq" Nov 02 14:55:04.060051 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.060021 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-be6018-dockercfg-gcwsw\"" Nov 02 14:55:04.068329 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.068299 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq"] Nov 02 14:55:04.075591 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.075560 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-be6018\"" Nov 02 14:55:04.125561 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.125522 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7"] Nov 02 14:55:04.129730 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.129700 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7" Nov 02 14:55:04.152604 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.152572 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7"] Nov 02 14:55:04.201787 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.201748 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd59de8e-8870-4843-8a4f-158ca0e775fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq\" (UID: \"dd59de8e-8870-4843-8a4f-158ca0e775fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq" Nov 02 14:55:04.202032 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.201963 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5xn2\" (UniqueName: \"kubernetes.io/projected/dd59de8e-8870-4843-8a4f-158ca0e775fb-kube-api-access-x5xn2\") pod \"payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq\" (UID: \"dd59de8e-8870-4843-8a4f-158ca0e775fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq" Nov 02 14:55:04.303569 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.303473 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzsg8\" (UniqueName: \"kubernetes.io/projected/f336edce-b7cb-47f7-8b99-6df1ec6e421d-kube-api-access-bzsg8\") pod \"payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7\" (UID: \"f336edce-b7cb-47f7-8b99-6df1ec6e421d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7" Nov 02 14:55:04.303569 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.303518 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f336edce-b7cb-47f7-8b99-6df1ec6e421d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7\" (UID: \"f336edce-b7cb-47f7-8b99-6df1ec6e421d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7" Nov 02 14:55:04.303768 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.303583 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5xn2\" (UniqueName: \"kubernetes.io/projected/dd59de8e-8870-4843-8a4f-158ca0e775fb-kube-api-access-x5xn2\") pod \"payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq\" (UID: \"dd59de8e-8870-4843-8a4f-158ca0e775fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq" Nov 02 14:55:04.303768 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.303637 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd59de8e-8870-4843-8a4f-158ca0e775fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq\" (UID: \"dd59de8e-8870-4843-8a4f-158ca0e775fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq" Nov 02 14:55:04.304365 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.304343 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd59de8e-8870-4843-8a4f-158ca0e775fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq\" (UID: \"dd59de8e-8870-4843-8a4f-158ca0e775fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq" Nov 02 14:55:04.327061 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.327022 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5xn2\" (UniqueName: \"kubernetes.io/projected/dd59de8e-8870-4843-8a4f-158ca0e775fb-kube-api-access-x5xn2\") pod \"payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq\" (UID: \"dd59de8e-8870-4843-8a4f-158ca0e775fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq" Nov 02 14:55:04.344597 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.344565 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq" Nov 02 14:55:04.404810 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.404777 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzsg8\" (UniqueName: \"kubernetes.io/projected/f336edce-b7cb-47f7-8b99-6df1ec6e421d-kube-api-access-bzsg8\") pod \"payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7\" (UID: \"f336edce-b7cb-47f7-8b99-6df1ec6e421d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7" Nov 02 14:55:04.404942 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.404818 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f336edce-b7cb-47f7-8b99-6df1ec6e421d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7\" (UID: \"f336edce-b7cb-47f7-8b99-6df1ec6e421d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7" Nov 02 14:55:04.405723 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.405704 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f336edce-b7cb-47f7-8b99-6df1ec6e421d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7\" (UID: \"f336edce-b7cb-47f7-8b99-6df1ec6e421d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7" Nov 02 14:55:04.428531 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.428495 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzsg8\" (UniqueName: \"kubernetes.io/projected/f336edce-b7cb-47f7-8b99-6df1ec6e421d-kube-api-access-bzsg8\") pod \"payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7\" (UID: \"f336edce-b7cb-47f7-8b99-6df1ec6e421d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7" Nov 02 14:55:04.443775 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.443745 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7" Nov 02 14:55:04.643833 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.643570 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq"] Nov 02 14:55:04.649718 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:55:04.649681 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd59de8e_8870_4843_8a4f_158ca0e775fb.slice/crio-269bd3a4369390f5a39865e642087810a0abe8030d568ebd7c7e988133d18f43 WatchSource:0}: Error finding container 269bd3a4369390f5a39865e642087810a0abe8030d568ebd7c7e988133d18f43: Status 404 returned error can't find the container with id 269bd3a4369390f5a39865e642087810a0abe8030d568ebd7c7e988133d18f43 Nov 02 14:55:04.746406 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:04.746343 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7"] Nov 02 14:55:04.749331 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:55:04.749300 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf336edce_b7cb_47f7_8b99_6df1ec6e421d.slice/crio-90f851e2461b998dbd703b87db7f5324c52d4b3720a787349fc167384ea9e51c WatchSource:0}: Error finding container 90f851e2461b998dbd703b87db7f5324c52d4b3720a787349fc167384ea9e51c: Status 404 returned error can't find the container with id 90f851e2461b998dbd703b87db7f5324c52d4b3720a787349fc167384ea9e51c Nov 02 14:55:05.647565 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:05.647523 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7" event={"ID":"f336edce-b7cb-47f7-8b99-6df1ec6e421d","Type":"ContainerStarted","Data":"9ba804950549f40f591ecc6501165aec7c3c2ed24af86db6230a856b25ba9109"} Nov 02 14:55:05.648075 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:05.647589 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7" event={"ID":"f336edce-b7cb-47f7-8b99-6df1ec6e421d","Type":"ContainerStarted","Data":"90f851e2461b998dbd703b87db7f5324c52d4b3720a787349fc167384ea9e51c"} Nov 02 14:55:05.648075 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:05.647706 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7" Nov 02 14:55:05.649103 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:05.649057 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq" event={"ID":"dd59de8e-8870-4843-8a4f-158ca0e775fb","Type":"ContainerStarted","Data":"319fe48082a7fdfef29db52262bac9b1799610f22337aae8d4741b84ed937135"} Nov 02 14:55:05.649103 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:05.649104 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq" Nov 02 14:55:05.649311 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:05.649126 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq" event={"ID":"dd59de8e-8870-4843-8a4f-158ca0e775fb","Type":"ContainerStarted","Data":"269bd3a4369390f5a39865e642087810a0abe8030d568ebd7c7e988133d18f43"} Nov 02 14:55:05.687841 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:05.687786 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7" podStartSLOduration=1.687773067 podStartE2EDuration="1.687773067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:05.686395706 +0000 UTC m=+3470.870243054" watchObservedRunningTime="2026-11-02 14:55:05.687773067 +0000 UTC m=+3470.871620404" Nov 02 14:55:05.727777 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:05.727723 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq" podStartSLOduration=1.7277094979999998 podStartE2EDuration="1.727709498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:05.726025744 +0000 UTC m=+3470.909873093" watchObservedRunningTime="2026-11-02 14:55:05.727709498 +0000 UTC m=+3470.911556834" Nov 02 14:55:06.010180 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:55:06.010140 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:55:16.657825 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:16.657745 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq" Nov 02 14:55:16.657825 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:16.657816 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7" Nov 02 14:55:19.720216 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:19.720179 4866 generic.go:356] "Generic (PLEG): container finished" podID="9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" containerID="a49e97c5284865666d4eb221665e836a2d09a9cdc8ea21f600a8b7bb33e2837e" exitCode=0 Nov 02 14:55:19.720684 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:19.720256 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" event={"ID":"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4","Type":"ContainerDied","Data":"a49e97c5284865666d4eb221665e836a2d09a9cdc8ea21f600a8b7bb33e2837e"} Nov 02 14:55:19.720729 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:19.720685 4866 scope.go:122] "RemoveContainer" containerID="a49e97c5284865666d4eb221665e836a2d09a9cdc8ea21f600a8b7bb33e2837e" Nov 02 14:55:19.722001 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:19.721972 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:55:19.722080 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:19.722039 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:55:19.722119 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:19.722082 4866 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 02 14:55:20.731969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:20.731913 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" event={"ID":"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4","Type":"ContainerStarted","Data":"2f4d904544ab74401203fe2f1c73154d27349f3ec8973948621dffd596bb00c9"} Nov 02 14:55:21.250165 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:21.250128 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:55:21.250379 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:21.250359 4866 patch_prober.go:29] interesting pod/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 02 14:55:21.250439 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:21.250418 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" podUID="9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 02 14:55:22.231131 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:22.231092 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7"] Nov 02 14:55:22.250298 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:22.250261 4866 patch_prober.go:29] interesting pod/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 02 14:55:22.250496 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:22.250330 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" podUID="9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 02 14:55:22.268024 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:22.267995 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq"] Nov 02 14:55:23.250332 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:23.250291 4866 patch_prober.go:29] interesting pod/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 02 14:55:23.250758 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:23.250351 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" podUID="9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 02 14:55:24.249421 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:24.249384 4866 patch_prober.go:29] interesting pod/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 02 14:55:24.249608 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:24.249442 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" podUID="9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 02 14:55:25.250325 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:25.250285 4866 patch_prober.go:29] interesting pod/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 02 14:55:25.250738 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:25.250345 4866 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" podUID="9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 02 14:55:25.905307 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:25.905267 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9"] Nov 02 14:55:25.909265 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:25.909238 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9" Nov 02 14:55:25.934930 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:25.934888 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-be6018-dockercfg-6zvgh\"" Nov 02 14:55:25.937861 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:25.937831 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9"] Nov 02 14:55:26.058887 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.058852 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s"] Nov 02 14:55:26.063279 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.063255 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s" Nov 02 14:55:26.067660 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.067616 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2d2400c-5420-4198-9af1-7e54818595bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9\" (UID: \"a2d2400c-5420-4198-9af1-7e54818595bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9" Nov 02 14:55:26.067983 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.067938 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxl9h\" (UniqueName: \"kubernetes.io/projected/a2d2400c-5420-4198-9af1-7e54818595bd-kube-api-access-fxl9h\") pod \"payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9\" (UID: \"a2d2400c-5420-4198-9af1-7e54818595bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9" Nov 02 14:55:26.094004 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.093970 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s"] Nov 02 14:55:26.168630 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.168548 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jw2q\" (UniqueName: \"kubernetes.io/projected/13b35ec4-565c-4b88-867d-638252310831-kube-api-access-7jw2q\") pod \"payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s\" (UID: \"13b35ec4-565c-4b88-867d-638252310831\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s" Nov 02 14:55:26.168630 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.168618 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxl9h\" (UniqueName: \"kubernetes.io/projected/a2d2400c-5420-4198-9af1-7e54818595bd-kube-api-access-fxl9h\") pod \"payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9\" (UID: \"a2d2400c-5420-4198-9af1-7e54818595bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9" Nov 02 14:55:26.168821 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.168773 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b35ec4-565c-4b88-867d-638252310831-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s\" (UID: \"13b35ec4-565c-4b88-867d-638252310831\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s" Nov 02 14:55:26.168859 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.168821 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2d2400c-5420-4198-9af1-7e54818595bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9\" (UID: \"a2d2400c-5420-4198-9af1-7e54818595bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9" Nov 02 14:55:26.169533 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.169514 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2d2400c-5420-4198-9af1-7e54818595bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9\" (UID: \"a2d2400c-5420-4198-9af1-7e54818595bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9" Nov 02 14:55:26.192527 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.192495 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxl9h\" (UniqueName: \"kubernetes.io/projected/a2d2400c-5420-4198-9af1-7e54818595bd-kube-api-access-fxl9h\") pod \"payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9\" (UID: \"a2d2400c-5420-4198-9af1-7e54818595bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9" Nov 02 14:55:26.222651 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.222612 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9" Nov 02 14:55:26.254048 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.254018 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:55:26.254563 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.254384 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:55:26.255189 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.255137 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:55:26.269765 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.269730 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jw2q\" (UniqueName: \"kubernetes.io/projected/13b35ec4-565c-4b88-867d-638252310831-kube-api-access-7jw2q\") pod \"payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s\" (UID: \"13b35ec4-565c-4b88-867d-638252310831\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s" Nov 02 14:55:26.269934 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.269860 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b35ec4-565c-4b88-867d-638252310831-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s\" (UID: \"13b35ec4-565c-4b88-867d-638252310831\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s" Nov 02 14:55:26.270739 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.270715 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b35ec4-565c-4b88-867d-638252310831-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s\" (UID: \"13b35ec4-565c-4b88-867d-638252310831\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s" Nov 02 14:55:26.294632 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.294598 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jw2q\" (UniqueName: \"kubernetes.io/projected/13b35ec4-565c-4b88-867d-638252310831-kube-api-access-7jw2q\") pod \"payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s\" (UID: \"13b35ec4-565c-4b88-867d-638252310831\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s" Nov 02 14:55:26.376701 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.376668 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s" Nov 02 14:55:26.522596 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.522558 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9"] Nov 02 14:55:26.524294 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:55:26.524270 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d2400c_5420_4198_9af1_7e54818595bd.slice/crio-d9eee155e00c68f544fce2bf9a52e485f63e13dce961d5fc1ca390f3a11ce2b1 WatchSource:0}: Error finding container d9eee155e00c68f544fce2bf9a52e485f63e13dce961d5fc1ca390f3a11ce2b1: Status 404 returned error can't find the container with id d9eee155e00c68f544fce2bf9a52e485f63e13dce961d5fc1ca390f3a11ce2b1 Nov 02 14:55:26.526153 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.526139 4866 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:55:26.689392 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.689359 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s"] Nov 02 14:55:26.691536 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:55:26.691511 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b35ec4_565c_4b88_867d_638252310831.slice/crio-674bcf8f1138a7243db13e8e580515aef05f735ecbb66cc397399533daaf16df WatchSource:0}: Error finding container 674bcf8f1138a7243db13e8e580515aef05f735ecbb66cc397399533daaf16df: Status 404 returned error can't find the container with id 674bcf8f1138a7243db13e8e580515aef05f735ecbb66cc397399533daaf16df Nov 02 14:55:26.767456 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.767419 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s" event={"ID":"13b35ec4-565c-4b88-867d-638252310831","Type":"ContainerStarted","Data":"674bcf8f1138a7243db13e8e580515aef05f735ecbb66cc397399533daaf16df"} Nov 02 14:55:26.769758 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:26.769719 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9" event={"ID":"a2d2400c-5420-4198-9af1-7e54818595bd","Type":"ContainerStarted","Data":"d9eee155e00c68f544fce2bf9a52e485f63e13dce961d5fc1ca390f3a11ce2b1"} Nov 02 14:55:27.231830 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.231782 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7" podUID="f336edce-b7cb-47f7-8b99-6df1ec6e421d" containerName="payload-processing" containerID="cri-o://9ba804950549f40f591ecc6501165aec7c3c2ed24af86db6230a856b25ba9109" gracePeriod=25 Nov 02 14:55:27.268671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.268595 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq" podUID="dd59de8e-8870-4843-8a4f-158ca0e775fb" containerName="payload-pre-processing" containerID="cri-o://319fe48082a7fdfef29db52262bac9b1799610f22337aae8d4741b84ed937135" gracePeriod=25 Nov 02 14:55:27.584748 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.584428 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7" Nov 02 14:55:27.621178 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.621147 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq" Nov 02 14:55:27.686857 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.686821 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f336edce-b7cb-47f7-8b99-6df1ec6e421d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f336edce-b7cb-47f7-8b99-6df1ec6e421d-plugins-config-volume\") pod \"f336edce-b7cb-47f7-8b99-6df1ec6e421d\" (UID: \"f336edce-b7cb-47f7-8b99-6df1ec6e421d\") " Nov 02 14:55:27.687099 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.686901 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f336edce-b7cb-47f7-8b99-6df1ec6e421d-kube-api-access-bzsg8\" (UniqueName: \"kubernetes.io/projected/f336edce-b7cb-47f7-8b99-6df1ec6e421d-kube-api-access-bzsg8\") pod \"f336edce-b7cb-47f7-8b99-6df1ec6e421d\" (UID: \"f336edce-b7cb-47f7-8b99-6df1ec6e421d\") " Nov 02 14:55:27.687351 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.687324 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f336edce-b7cb-47f7-8b99-6df1ec6e421d-plugins-config-volume" pod "f336edce-b7cb-47f7-8b99-6df1ec6e421d" (UID: "f336edce-b7cb-47f7-8b99-6df1ec6e421d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:55:27.689471 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.689440 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f336edce-b7cb-47f7-8b99-6df1ec6e421d-kube-api-access-bzsg8" pod "f336edce-b7cb-47f7-8b99-6df1ec6e421d" (UID: "f336edce-b7cb-47f7-8b99-6df1ec6e421d"). InnerVolumeSpecName "kube-api-access-bzsg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:55:27.779535 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.779189 4866 generic.go:356] "Generic (PLEG): container finished" podID="f336edce-b7cb-47f7-8b99-6df1ec6e421d" containerID="9ba804950549f40f591ecc6501165aec7c3c2ed24af86db6230a856b25ba9109" exitCode=0 Nov 02 14:55:27.779535 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.779251 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7" Nov 02 14:55:27.779535 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.779307 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7" event={"ID":"f336edce-b7cb-47f7-8b99-6df1ec6e421d","Type":"ContainerDied","Data":"9ba804950549f40f591ecc6501165aec7c3c2ed24af86db6230a856b25ba9109"} Nov 02 14:55:27.779535 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.779350 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7" event={"ID":"f336edce-b7cb-47f7-8b99-6df1ec6e421d","Type":"ContainerDied","Data":"90f851e2461b998dbd703b87db7f5324c52d4b3720a787349fc167384ea9e51c"} Nov 02 14:55:27.779535 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.779374 4866 scope.go:122] "RemoveContainer" containerID="9ba804950549f40f591ecc6501165aec7c3c2ed24af86db6230a856b25ba9109" Nov 02 14:55:27.785799 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.784690 4866 generic.go:356] "Generic (PLEG): container finished" podID="dd59de8e-8870-4843-8a4f-158ca0e775fb" containerID="319fe48082a7fdfef29db52262bac9b1799610f22337aae8d4741b84ed937135" exitCode=0 Nov 02 14:55:27.785799 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.785581 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq" Nov 02 14:55:27.785799 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.785710 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq" event={"ID":"dd59de8e-8870-4843-8a4f-158ca0e775fb","Type":"ContainerDied","Data":"319fe48082a7fdfef29db52262bac9b1799610f22337aae8d4741b84ed937135"} Nov 02 14:55:27.785799 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.785752 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq" event={"ID":"dd59de8e-8870-4843-8a4f-158ca0e775fb","Type":"ContainerDied","Data":"269bd3a4369390f5a39865e642087810a0abe8030d568ebd7c7e988133d18f43"} Nov 02 14:55:27.787693 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.787664 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd59de8e-8870-4843-8a4f-158ca0e775fb-kube-api-access-x5xn2\" (UniqueName: \"kubernetes.io/projected/dd59de8e-8870-4843-8a4f-158ca0e775fb-kube-api-access-x5xn2\") pod \"dd59de8e-8870-4843-8a4f-158ca0e775fb\" (UID: \"dd59de8e-8870-4843-8a4f-158ca0e775fb\") " Nov 02 14:55:27.787838 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.787716 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd59de8e-8870-4843-8a4f-158ca0e775fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd59de8e-8870-4843-8a4f-158ca0e775fb-plugins-config-volume\") pod \"dd59de8e-8870-4843-8a4f-158ca0e775fb\" (UID: \"dd59de8e-8870-4843-8a4f-158ca0e775fb\") " Nov 02 14:55:27.788229 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.788181 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f336edce-b7cb-47f7-8b99-6df1ec6e421d-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:55:27.788229 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.788207 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzsg8\" (UniqueName: \"kubernetes.io/projected/f336edce-b7cb-47f7-8b99-6df1ec6e421d-kube-api-access-bzsg8\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:55:27.788405 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.788293 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd59de8e-8870-4843-8a4f-158ca0e775fb-plugins-config-volume" pod "dd59de8e-8870-4843-8a4f-158ca0e775fb" (UID: "dd59de8e-8870-4843-8a4f-158ca0e775fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:55:27.791355 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.791316 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd59de8e-8870-4843-8a4f-158ca0e775fb-kube-api-access-x5xn2" pod "dd59de8e-8870-4843-8a4f-158ca0e775fb" (UID: "dd59de8e-8870-4843-8a4f-158ca0e775fb"). InnerVolumeSpecName "kube-api-access-x5xn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:55:27.795217 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.795189 4866 scope.go:122] "RemoveContainer" containerID="9ba804950549f40f591ecc6501165aec7c3c2ed24af86db6230a856b25ba9109" Nov 02 14:55:27.796083 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:55:27.796050 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba804950549f40f591ecc6501165aec7c3c2ed24af86db6230a856b25ba9109\": container with ID starting with 9ba804950549f40f591ecc6501165aec7c3c2ed24af86db6230a856b25ba9109 not found: ID does not exist" containerID="9ba804950549f40f591ecc6501165aec7c3c2ed24af86db6230a856b25ba9109" Nov 02 14:55:27.796191 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.796092 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba804950549f40f591ecc6501165aec7c3c2ed24af86db6230a856b25ba9109"} err="failed to get container status \"9ba804950549f40f591ecc6501165aec7c3c2ed24af86db6230a856b25ba9109\": rpc error: code = NotFound desc = could not find container \"9ba804950549f40f591ecc6501165aec7c3c2ed24af86db6230a856b25ba9109\": container with ID starting with 9ba804950549f40f591ecc6501165aec7c3c2ed24af86db6230a856b25ba9109 not found: ID does not exist" Nov 02 14:55:27.796191 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.796115 4866 scope.go:122] "RemoveContainer" containerID="319fe48082a7fdfef29db52262bac9b1799610f22337aae8d4741b84ed937135" Nov 02 14:55:27.808979 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.808934 4866 scope.go:122] "RemoveContainer" containerID="319fe48082a7fdfef29db52262bac9b1799610f22337aae8d4741b84ed937135" Nov 02 14:55:27.809318 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:55:27.809293 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319fe48082a7fdfef29db52262bac9b1799610f22337aae8d4741b84ed937135\": container with ID starting with 319fe48082a7fdfef29db52262bac9b1799610f22337aae8d4741b84ed937135 not found: ID does not exist" containerID="319fe48082a7fdfef29db52262bac9b1799610f22337aae8d4741b84ed937135" Nov 02 14:55:27.809421 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.809328 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319fe48082a7fdfef29db52262bac9b1799610f22337aae8d4741b84ed937135"} err="failed to get container status \"319fe48082a7fdfef29db52262bac9b1799610f22337aae8d4741b84ed937135\": rpc error: code = NotFound desc = could not find container \"319fe48082a7fdfef29db52262bac9b1799610f22337aae8d4741b84ed937135\": container with ID starting with 319fe48082a7fdfef29db52262bac9b1799610f22337aae8d4741b84ed937135 not found: ID does not exist" Nov 02 14:55:27.848089 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.848052 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7"] Nov 02 14:55:27.852498 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.852461 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6018-66d69866c-vlww7"] Nov 02 14:55:27.889384 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.889343 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5xn2\" (UniqueName: \"kubernetes.io/projected/dd59de8e-8870-4843-8a4f-158ca0e775fb-kube-api-access-x5xn2\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:55:27.889384 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:27.889380 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd59de8e-8870-4843-8a4f-158ca0e775fb-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:55:28.134652 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:28.134610 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq"] Nov 02 14:55:28.138533 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:28.138505 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-58f558f76c-h5mbq"] Nov 02 14:55:28.793446 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:28.793408 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s" event={"ID":"13b35ec4-565c-4b88-867d-638252310831","Type":"ContainerStarted","Data":"fcbe28e52621ae5beec59d62fa4292c6a41f942b2f240b43f606595f0f072f43"} Nov 02 14:55:28.794032 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:28.793521 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s" Nov 02 14:55:28.795160 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:28.795137 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9" event={"ID":"a2d2400c-5420-4198-9af1-7e54818595bd","Type":"ContainerStarted","Data":"1b749e5b8039908924427ff360b6404711128e8112455440d1b6aaf14b398619"} Nov 02 14:55:28.795274 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:28.795191 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9" Nov 02 14:55:28.835260 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:28.835202 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s" podStartSLOduration=1.06922798 podStartE2EDuration="2.83518569s" totalImagesPullingTime="1.76595771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:28.833390844 +0000 UTC m=+3494.017238181" watchObservedRunningTime="2026-11-02 14:55:28.83518569 +0000 UTC m=+3494.019033026" Nov 02 14:55:28.874252 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:28.874187 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9" podStartSLOduration=1.993451152 podStartE2EDuration="3.874172596s" totalImagesPullingTime="1.880721444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:28.873148697 +0000 UTC m=+3494.056996034" watchObservedRunningTime="2026-11-02 14:55:28.874172596 +0000 UTC m=+3494.058019932" Nov 02 14:55:28.944878 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:28.944847 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd59de8e-8870-4843-8a4f-158ca0e775fb" path="/var/lib/kubelet/pods/dd59de8e-8870-4843-8a4f-158ca0e775fb/volumes" Nov 02 14:55:28.945290 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:28.945275 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f336edce-b7cb-47f7-8b99-6df1ec6e421d" path="/var/lib/kubelet/pods/f336edce-b7cb-47f7-8b99-6df1ec6e421d/volumes" Nov 02 14:55:36.008738 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:55:36.008699 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:55:39.803417 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:39.803380 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s" Nov 02 14:55:39.803815 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:39.803457 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9" Nov 02 14:55:49.796963 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:49.796926 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rx5hk_ce41c5ab-a5da-44b5-bf3e-f045c70f55ce/payload-processing/0.log" Nov 02 14:55:50.004177 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:50.004147 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rx5hk_ce41c5ab-a5da-44b5-bf3e-f045c70f55ce/payload-processing/0.log" Nov 02 14:55:50.207526 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:50.207496 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s_13b35ec4-565c-4b88-867d-638252310831/payload-processing/0.log" Nov 02 14:55:52.676601 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.676561 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv"] Nov 02 14:55:52.677171 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.677150 4866 cpu_manager.go:414] "containerMap: removing container" podUID="f336edce-b7cb-47f7-8b99-6df1ec6e421d" containerName="payload-processing" Nov 02 14:55:52.677171 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.677171 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f336edce-b7cb-47f7-8b99-6df1ec6e421d" containerName="payload-processing" Nov 02 14:55:52.677293 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.677200 4866 cpu_manager.go:414] "containerMap: removing container" podUID="dd59de8e-8870-4843-8a4f-158ca0e775fb" containerName="payload-pre-processing" Nov 02 14:55:52.677293 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.677206 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd59de8e-8870-4843-8a4f-158ca0e775fb" containerName="payload-pre-processing" Nov 02 14:55:52.677391 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.677296 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="f336edce-b7cb-47f7-8b99-6df1ec6e421d" containerName="payload-processing" Nov 02 14:55:52.677391 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.677307 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd59de8e-8870-4843-8a4f-158ca0e775fb" containerName="payload-pre-processing" Nov 02 14:55:52.680360 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.680341 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:52.706383 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.706345 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b21aae\"/\"default-dockercfg-c5kjx\"" Nov 02 14:55:52.706383 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.706368 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b21aae\"/\"kube-root-ca.crt\"" Nov 02 14:55:52.706383 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.706347 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b21aae\"/\"openshift-service-ca.crt\"" Nov 02 14:55:52.706678 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.706446 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b21aae\"/\"ipp-route-b21aae-kserve-self-signed-certs\"" Nov 02 14:55:52.708571 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.708547 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv"] Nov 02 14:55:52.729563 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.729525 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-dshm\") pod \"ipp-route-b21aae-kserve-579b5776c7-pz6bv\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:52.729752 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.729571 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdbtn\" (UniqueName: \"kubernetes.io/projected/343fe571-5a17-4108-947b-c393031b192d-kube-api-access-wdbtn\") pod \"ipp-route-b21aae-kserve-579b5776c7-pz6bv\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:52.729752 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.729664 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-model-cache\") pod \"ipp-route-b21aae-kserve-579b5776c7-pz6bv\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:52.729861 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.729759 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-tmp-dir\") pod \"ipp-route-b21aae-kserve-579b5776c7-pz6bv\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:52.729861 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.729794 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-home\") pod \"ipp-route-b21aae-kserve-579b5776c7-pz6bv\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:52.729967 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.729897 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343fe571-5a17-4108-947b-c393031b192d-tls-certs\") pod \"ipp-route-b21aae-kserve-579b5776c7-pz6bv\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:52.831289 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.831244 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-dshm\") pod \"ipp-route-b21aae-kserve-579b5776c7-pz6bv\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:52.831289 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.831295 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdbtn\" (UniqueName: \"kubernetes.io/projected/343fe571-5a17-4108-947b-c393031b192d-kube-api-access-wdbtn\") pod \"ipp-route-b21aae-kserve-579b5776c7-pz6bv\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:52.831580 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.831319 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-model-cache\") pod \"ipp-route-b21aae-kserve-579b5776c7-pz6bv\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:52.831580 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.831356 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-tmp-dir\") pod \"ipp-route-b21aae-kserve-579b5776c7-pz6bv\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:52.831580 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.831372 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-home\") pod \"ipp-route-b21aae-kserve-579b5776c7-pz6bv\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:52.831580 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.831419 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343fe571-5a17-4108-947b-c393031b192d-tls-certs\") pod \"ipp-route-b21aae-kserve-579b5776c7-pz6bv\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:52.831834 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.831809 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-model-cache\") pod \"ipp-route-b21aae-kserve-579b5776c7-pz6bv\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:52.831875 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.831829 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-tmp-dir\") pod \"ipp-route-b21aae-kserve-579b5776c7-pz6bv\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:52.831918 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.831897 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-home\") pod \"ipp-route-b21aae-kserve-579b5776c7-pz6bv\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:52.833654 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.833629 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-dshm\") pod \"ipp-route-b21aae-kserve-579b5776c7-pz6bv\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:52.833927 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.833909 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343fe571-5a17-4108-947b-c393031b192d-tls-certs\") pod \"ipp-route-b21aae-kserve-579b5776c7-pz6bv\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:52.855938 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.855904 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdbtn\" (UniqueName: \"kubernetes.io/projected/343fe571-5a17-4108-947b-c393031b192d-kube-api-access-wdbtn\") pod \"ipp-route-b21aae-kserve-579b5776c7-pz6bv\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:52.993225 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:52.993185 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:55:53.294842 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:53.294788 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv"] Nov 02 14:55:53.297425 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:55:53.297396 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod343fe571_5a17_4108_947b_c393031b192d.slice/crio-d1ceb9a051075bd494b6a58e0325e5e7fe6093a3652f5e12a5ee3a664326599c WatchSource:0}: Error finding container d1ceb9a051075bd494b6a58e0325e5e7fe6093a3652f5e12a5ee3a664326599c: Status 404 returned error can't find the container with id d1ceb9a051075bd494b6a58e0325e5e7fe6093a3652f5e12a5ee3a664326599c Nov 02 14:55:53.931295 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:53.931256 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" event={"ID":"343fe571-5a17-4108-947b-c393031b192d","Type":"ContainerStarted","Data":"a579c8ec52caf3a58d86bccc7759262d88d653dcc3e968f68d8fb688342fbb45"} Nov 02 14:55:53.931295 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:53.931297 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" event={"ID":"343fe571-5a17-4108-947b-c393031b192d","Type":"ContainerStarted","Data":"d1ceb9a051075bd494b6a58e0325e5e7fe6093a3652f5e12a5ee3a664326599c"} Nov 02 14:55:53.973816 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:53.973759 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" podStartSLOduration=1.973746329 podStartE2EDuration="1.973746329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:53.972020292 +0000 UTC m=+3519.155867629" watchObservedRunningTime="2026-11-02 14:55:53.973746329 +0000 UTC m=+3519.157593708" Nov 02 14:55:53.993308 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:55:53.993273 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:56:02.993321 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:56:02.993288 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:56:03.009739 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:56:03.009704 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:56:04.005277 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:56:04.005242 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:56:06.011038 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:56:06.010994 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:56:16.453196 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:56:16.453164 4866 scope.go:122] "RemoveContainer" containerID="91d7f0fb02e6f6f6ee7f248e7656399fe399762e652028cbb34565755d8e9514" Nov 02 14:56:34.277496 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:56:34.277463 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rx5hk_ce41c5ab-a5da-44b5-bf3e-f045c70f55ce/payload-processing/0.log" Nov 02 14:56:34.486126 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:56:34.486095 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rx5hk_ce41c5ab-a5da-44b5-bf3e-f045c70f55ce/payload-processing/0.log" Nov 02 14:56:34.688009 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:56:34.687913 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s_13b35ec4-565c-4b88-867d-638252310831/payload-processing/0.log" Nov 02 14:56:34.885139 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:56:34.885109 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s_13b35ec4-565c-4b88-867d-638252310831/payload-processing/0.log" Nov 02 14:56:36.010105 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:56:36.010069 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:56:40.656924 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:56:40.656858 4866 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" podUID="fcd82cc88301759f1e08e2b014a50651" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:56:53.058809 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:56:53.058766 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv"] Nov 02 14:57:06.010633 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:57:06.010595 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:57:08.079192 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.079143 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" podUID="343fe571-5a17-4108-947b-c393031b192d" containerName="main" containerID="cri-o://a579c8ec52caf3a58d86bccc7759262d88d653dcc3e968f68d8fb688342fbb45" gracePeriod=45 Nov 02 14:57:08.333671 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.333601 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:57:08.338962 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.338920 4866 generic.go:356] "Generic (PLEG): container finished" podID="343fe571-5a17-4108-947b-c393031b192d" containerID="a579c8ec52caf3a58d86bccc7759262d88d653dcc3e968f68d8fb688342fbb45" exitCode=0 Nov 02 14:57:08.339114 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.338984 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" Nov 02 14:57:08.339114 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.338978 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" event={"ID":"343fe571-5a17-4108-947b-c393031b192d","Type":"ContainerDied","Data":"a579c8ec52caf3a58d86bccc7759262d88d653dcc3e968f68d8fb688342fbb45"} Nov 02 14:57:08.339114 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.339103 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv" event={"ID":"343fe571-5a17-4108-947b-c393031b192d","Type":"ContainerDied","Data":"d1ceb9a051075bd494b6a58e0325e5e7fe6093a3652f5e12a5ee3a664326599c"} Nov 02 14:57:08.339219 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.339124 4866 scope.go:122] "RemoveContainer" containerID="a579c8ec52caf3a58d86bccc7759262d88d653dcc3e968f68d8fb688342fbb45" Nov 02 14:57:08.347808 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.347789 4866 scope.go:122] "RemoveContainer" containerID="a579c8ec52caf3a58d86bccc7759262d88d653dcc3e968f68d8fb688342fbb45" Nov 02 14:57:08.348152 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:57:08.348127 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a579c8ec52caf3a58d86bccc7759262d88d653dcc3e968f68d8fb688342fbb45\": container with ID starting with a579c8ec52caf3a58d86bccc7759262d88d653dcc3e968f68d8fb688342fbb45 not found: ID does not exist" containerID="a579c8ec52caf3a58d86bccc7759262d88d653dcc3e968f68d8fb688342fbb45" Nov 02 14:57:08.348226 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.348166 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a579c8ec52caf3a58d86bccc7759262d88d653dcc3e968f68d8fb688342fbb45"} err="failed to get container status \"a579c8ec52caf3a58d86bccc7759262d88d653dcc3e968f68d8fb688342fbb45\": rpc error: code = NotFound desc = could not find container \"a579c8ec52caf3a58d86bccc7759262d88d653dcc3e968f68d8fb688342fbb45\": container with ID starting with a579c8ec52caf3a58d86bccc7759262d88d653dcc3e968f68d8fb688342fbb45 not found: ID does not exist" Nov 02 14:57:08.401094 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.401063 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-home\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-home\") pod \"343fe571-5a17-4108-947b-c393031b192d\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " Nov 02 14:57:08.401305 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.401117 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-tmp-dir\") pod \"343fe571-5a17-4108-947b-c393031b192d\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " Nov 02 14:57:08.401305 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.401193 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-dshm\") pod \"343fe571-5a17-4108-947b-c393031b192d\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " Nov 02 14:57:08.401305 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.401236 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/343fe571-5a17-4108-947b-c393031b192d-tls-certs\" (UniqueName: \"kubernetes.io/secret/343fe571-5a17-4108-947b-c393031b192d-tls-certs\") pod \"343fe571-5a17-4108-947b-c393031b192d\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " Nov 02 14:57:08.401305 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.401299 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-model-cache\") pod \"343fe571-5a17-4108-947b-c393031b192d\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " Nov 02 14:57:08.401615 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.401436 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-home" pod "343fe571-5a17-4108-947b-c393031b192d" (UID: "343fe571-5a17-4108-947b-c393031b192d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:08.401615 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.401484 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/343fe571-5a17-4108-947b-c393031b192d-kube-api-access-wdbtn\" (UniqueName: \"kubernetes.io/projected/343fe571-5a17-4108-947b-c393031b192d-kube-api-access-wdbtn\") pod \"343fe571-5a17-4108-947b-c393031b192d\" (UID: \"343fe571-5a17-4108-947b-c393031b192d\") " Nov 02 14:57:08.401615 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.401502 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-tmp-dir" pod "343fe571-5a17-4108-947b-c393031b192d" (UID: "343fe571-5a17-4108-947b-c393031b192d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:08.401615 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.401573 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-model-cache" pod "343fe571-5a17-4108-947b-c393031b192d" (UID: "343fe571-5a17-4108-947b-c393031b192d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:08.401860 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.401842 4866 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-model-cache\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:57:08.401925 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.401862 4866 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-home\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:57:08.401925 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.401874 4866 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-tmp-dir\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:57:08.403729 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.403699 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-dshm" pod "343fe571-5a17-4108-947b-c393031b192d" (UID: "343fe571-5a17-4108-947b-c393031b192d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:08.403869 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.403841 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/343fe571-5a17-4108-947b-c393031b192d-tls-certs" pod "343fe571-5a17-4108-947b-c393031b192d" (UID: "343fe571-5a17-4108-947b-c393031b192d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:57:08.403933 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.403874 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343fe571-5a17-4108-947b-c393031b192d-kube-api-access-wdbtn" pod "343fe571-5a17-4108-947b-c393031b192d" (UID: "343fe571-5a17-4108-947b-c393031b192d"). InnerVolumeSpecName "kube-api-access-wdbtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:08.503094 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.503057 4866 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343fe571-5a17-4108-947b-c393031b192d-dshm\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:57:08.503094 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.503086 4866 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343fe571-5a17-4108-947b-c393031b192d-tls-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:57:08.503094 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.503098 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdbtn\" (UniqueName: \"kubernetes.io/projected/343fe571-5a17-4108-947b-c393031b192d-kube-api-access-wdbtn\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:57:08.683261 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.683226 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv"] Nov 02 14:57:08.694569 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.694532 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-b21aae/ipp-route-b21aae-kserve-579b5776c7-pz6bv"] Nov 02 14:57:08.945330 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:08.945301 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="343fe571-5a17-4108-947b-c393031b192d" path="/var/lib/kubelet/pods/343fe571-5a17-4108-947b-c393031b192d/volumes" Nov 02 14:57:15.669111 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:15.669079 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:57:15.691657 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:15.691626 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 14:57:36.009834 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:57:36.009797 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:57:43.574204 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:43.574166 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn"] Nov 02 14:57:43.574764 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:43.574398 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" podUID="09be7298-cfa0-4d2c-8535-40ff55ec6eba" containerName="maas-api" containerID="cri-o://07479cd58fbc8367e83081966c34ad672426acb2c8b7c055cb7151ff79ae1e49" gracePeriod=30 Nov 02 14:57:43.851737 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:43.851709 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" Nov 02 14:57:43.954564 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:43.954526 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09be7298-cfa0-4d2c-8535-40ff55ec6eba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09be7298-cfa0-4d2c-8535-40ff55ec6eba-maas-api-tls\") pod \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\" (UID: \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\") " Nov 02 14:57:43.954740 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:43.954651 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09be7298-cfa0-4d2c-8535-40ff55ec6eba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09be7298-cfa0-4d2c-8535-40ff55ec6eba-openshift-service-ca\") pod \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\" (UID: \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\") " Nov 02 14:57:43.954740 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:43.954727 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09be7298-cfa0-4d2c-8535-40ff55ec6eba-kube-api-access-7669q\" (UniqueName: \"kubernetes.io/projected/09be7298-cfa0-4d2c-8535-40ff55ec6eba-kube-api-access-7669q\") pod \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\" (UID: \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\") " Nov 02 14:57:43.954825 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:43.954785 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09be7298-cfa0-4d2c-8535-40ff55ec6eba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/09be7298-cfa0-4d2c-8535-40ff55ec6eba-metrics-certs\") pod \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\" (UID: \"09be7298-cfa0-4d2c-8535-40ff55ec6eba\") " Nov 02 14:57:43.955135 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:43.955103 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09be7298-cfa0-4d2c-8535-40ff55ec6eba-openshift-service-ca" pod "09be7298-cfa0-4d2c-8535-40ff55ec6eba" (UID: "09be7298-cfa0-4d2c-8535-40ff55ec6eba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:57:43.955451 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:43.955421 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09be7298-cfa0-4d2c-8535-40ff55ec6eba-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:57:43.956808 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:43.956781 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09be7298-cfa0-4d2c-8535-40ff55ec6eba-maas-api-tls" pod "09be7298-cfa0-4d2c-8535-40ff55ec6eba" (UID: "09be7298-cfa0-4d2c-8535-40ff55ec6eba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:57:43.956895 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:43.956869 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09be7298-cfa0-4d2c-8535-40ff55ec6eba-metrics-certs" pod "09be7298-cfa0-4d2c-8535-40ff55ec6eba" (UID: "09be7298-cfa0-4d2c-8535-40ff55ec6eba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:57:43.956997 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:43.956981 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09be7298-cfa0-4d2c-8535-40ff55ec6eba-kube-api-access-7669q" pod "09be7298-cfa0-4d2c-8535-40ff55ec6eba" (UID: "09be7298-cfa0-4d2c-8535-40ff55ec6eba"). InnerVolumeSpecName "kube-api-access-7669q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:44.056400 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:44.056367 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7669q\" (UniqueName: \"kubernetes.io/projected/09be7298-cfa0-4d2c-8535-40ff55ec6eba-kube-api-access-7669q\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:57:44.056400 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:44.056397 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09be7298-cfa0-4d2c-8535-40ff55ec6eba-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:57:44.056400 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:44.056407 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09be7298-cfa0-4d2c-8535-40ff55ec6eba-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:57:44.531621 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:44.531586 4866 generic.go:356] "Generic (PLEG): container finished" podID="09be7298-cfa0-4d2c-8535-40ff55ec6eba" containerID="07479cd58fbc8367e83081966c34ad672426acb2c8b7c055cb7151ff79ae1e49" exitCode=0 Nov 02 14:57:44.531811 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:44.531634 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" Nov 02 14:57:44.531811 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:44.531664 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" event={"ID":"09be7298-cfa0-4d2c-8535-40ff55ec6eba","Type":"ContainerDied","Data":"07479cd58fbc8367e83081966c34ad672426acb2c8b7c055cb7151ff79ae1e49"} Nov 02 14:57:44.531811 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:44.531777 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn" event={"ID":"09be7298-cfa0-4d2c-8535-40ff55ec6eba","Type":"ContainerDied","Data":"add0b1603ead0170198b60f4f83e9abfbbf4bd37a5f3f727228e8a3486354218"} Nov 02 14:57:44.531811 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:44.531804 4866 scope.go:122] "RemoveContainer" containerID="07479cd58fbc8367e83081966c34ad672426acb2c8b7c055cb7151ff79ae1e49" Nov 02 14:57:44.545488 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:44.545460 4866 scope.go:122] "RemoveContainer" containerID="07479cd58fbc8367e83081966c34ad672426acb2c8b7c055cb7151ff79ae1e49" Nov 02 14:57:44.546106 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:57:44.546060 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07479cd58fbc8367e83081966c34ad672426acb2c8b7c055cb7151ff79ae1e49\": container with ID starting with 07479cd58fbc8367e83081966c34ad672426acb2c8b7c055cb7151ff79ae1e49 not found: ID does not exist" containerID="07479cd58fbc8367e83081966c34ad672426acb2c8b7c055cb7151ff79ae1e49" Nov 02 14:57:44.546386 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:44.546338 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07479cd58fbc8367e83081966c34ad672426acb2c8b7c055cb7151ff79ae1e49"} err="failed to get container status \"07479cd58fbc8367e83081966c34ad672426acb2c8b7c055cb7151ff79ae1e49\": rpc error: code = NotFound desc = could not find container \"07479cd58fbc8367e83081966c34ad672426acb2c8b7c055cb7151ff79ae1e49\": container with ID starting with 07479cd58fbc8367e83081966c34ad672426acb2c8b7c055cb7151ff79ae1e49 not found: ID does not exist" Nov 02 14:57:44.590964 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:44.590912 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn"] Nov 02 14:57:44.597150 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:44.597116 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b21aae-6b59f5b955-h54fn"] Nov 02 14:57:44.944593 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:57:44.944551 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09be7298-cfa0-4d2c-8535-40ff55ec6eba" path="/var/lib/kubelet/pods/09be7298-cfa0-4d2c-8535-40ff55ec6eba/volumes" Nov 02 14:58:06.010117 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:58:06.010037 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:58:07.733450 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:07.733403 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s"] Nov 02 14:58:07.734118 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:07.733677 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s" podUID="13b35ec4-565c-4b88-867d-638252310831" containerName="payload-processing" containerID="cri-o://fcbe28e52621ae5beec59d62fa4292c6a41f942b2f240b43f606595f0f072f43" gracePeriod=30 Nov 02 14:58:07.763012 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:07.762977 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9"] Nov 02 14:58:07.763391 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:07.763336 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9" podUID="a2d2400c-5420-4198-9af1-7e54818595bd" containerName="payload-pre-processing" containerID="cri-o://1b749e5b8039908924427ff360b6404711128e8112455440d1b6aaf14b398619" gracePeriod=30 Nov 02 14:58:08.057989 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.055314 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s" Nov 02 14:58:08.059785 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.059620 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9" Nov 02 14:58:08.195740 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.195706 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2d2400c-5420-4198-9af1-7e54818595bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2d2400c-5420-4198-9af1-7e54818595bd-plugins-config-volume\") pod \"a2d2400c-5420-4198-9af1-7e54818595bd\" (UID: \"a2d2400c-5420-4198-9af1-7e54818595bd\") " Nov 02 14:58:08.195967 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.195761 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b35ec4-565c-4b88-867d-638252310831-kube-api-access-7jw2q\" (UniqueName: \"kubernetes.io/projected/13b35ec4-565c-4b88-867d-638252310831-kube-api-access-7jw2q\") pod \"13b35ec4-565c-4b88-867d-638252310831\" (UID: \"13b35ec4-565c-4b88-867d-638252310831\") " Nov 02 14:58:08.195967 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.195813 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2d2400c-5420-4198-9af1-7e54818595bd-kube-api-access-fxl9h\" (UniqueName: \"kubernetes.io/projected/a2d2400c-5420-4198-9af1-7e54818595bd-kube-api-access-fxl9h\") pod \"a2d2400c-5420-4198-9af1-7e54818595bd\" (UID: \"a2d2400c-5420-4198-9af1-7e54818595bd\") " Nov 02 14:58:08.195967 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.195891 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13b35ec4-565c-4b88-867d-638252310831-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b35ec4-565c-4b88-867d-638252310831-plugins-config-volume\") pod \"13b35ec4-565c-4b88-867d-638252310831\" (UID: \"13b35ec4-565c-4b88-867d-638252310831\") " Nov 02 14:58:08.196240 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.196215 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d2400c-5420-4198-9af1-7e54818595bd-plugins-config-volume" pod "a2d2400c-5420-4198-9af1-7e54818595bd" (UID: "a2d2400c-5420-4198-9af1-7e54818595bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:08.196385 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.196358 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b35ec4-565c-4b88-867d-638252310831-plugins-config-volume" pod "13b35ec4-565c-4b88-867d-638252310831" (UID: "13b35ec4-565c-4b88-867d-638252310831"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:08.198137 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.198111 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b35ec4-565c-4b88-867d-638252310831-kube-api-access-7jw2q" pod "13b35ec4-565c-4b88-867d-638252310831" (UID: "13b35ec4-565c-4b88-867d-638252310831"). InnerVolumeSpecName "kube-api-access-7jw2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:08.198244 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.198137 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d2400c-5420-4198-9af1-7e54818595bd-kube-api-access-fxl9h" pod "a2d2400c-5420-4198-9af1-7e54818595bd" (UID: "a2d2400c-5420-4198-9af1-7e54818595bd"). InnerVolumeSpecName "kube-api-access-fxl9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:08.297323 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.297231 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b35ec4-565c-4b88-867d-638252310831-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:58:08.297323 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.297268 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2d2400c-5420-4198-9af1-7e54818595bd-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:58:08.297323 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.297277 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jw2q\" (UniqueName: \"kubernetes.io/projected/13b35ec4-565c-4b88-867d-638252310831-kube-api-access-7jw2q\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:58:08.297323 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.297286 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxl9h\" (UniqueName: \"kubernetes.io/projected/a2d2400c-5420-4198-9af1-7e54818595bd-kube-api-access-fxl9h\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:58:08.652391 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.652297 4866 generic.go:356] "Generic (PLEG): container finished" podID="13b35ec4-565c-4b88-867d-638252310831" containerID="fcbe28e52621ae5beec59d62fa4292c6a41f942b2f240b43f606595f0f072f43" exitCode=0 Nov 02 14:58:08.652391 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.652346 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s" Nov 02 14:58:08.652622 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.652388 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s" event={"ID":"13b35ec4-565c-4b88-867d-638252310831","Type":"ContainerDied","Data":"fcbe28e52621ae5beec59d62fa4292c6a41f942b2f240b43f606595f0f072f43"} Nov 02 14:58:08.652622 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.652447 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s" event={"ID":"13b35ec4-565c-4b88-867d-638252310831","Type":"ContainerDied","Data":"674bcf8f1138a7243db13e8e580515aef05f735ecbb66cc397399533daaf16df"} Nov 02 14:58:08.652622 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.652479 4866 scope.go:122] "RemoveContainer" containerID="fcbe28e52621ae5beec59d62fa4292c6a41f942b2f240b43f606595f0f072f43" Nov 02 14:58:08.653803 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.653771 4866 generic.go:356] "Generic (PLEG): container finished" podID="a2d2400c-5420-4198-9af1-7e54818595bd" containerID="1b749e5b8039908924427ff360b6404711128e8112455440d1b6aaf14b398619" exitCode=0 Nov 02 14:58:08.653902 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.653801 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9" event={"ID":"a2d2400c-5420-4198-9af1-7e54818595bd","Type":"ContainerDied","Data":"1b749e5b8039908924427ff360b6404711128e8112455440d1b6aaf14b398619"} Nov 02 14:58:08.653902 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.653832 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9" Nov 02 14:58:08.653902 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.653844 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9" event={"ID":"a2d2400c-5420-4198-9af1-7e54818595bd","Type":"ContainerDied","Data":"d9eee155e00c68f544fce2bf9a52e485f63e13dce961d5fc1ca390f3a11ce2b1"} Nov 02 14:58:08.663138 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.663119 4866 scope.go:122] "RemoveContainer" containerID="fcbe28e52621ae5beec59d62fa4292c6a41f942b2f240b43f606595f0f072f43" Nov 02 14:58:08.663460 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:58:08.663440 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcbe28e52621ae5beec59d62fa4292c6a41f942b2f240b43f606595f0f072f43\": container with ID starting with fcbe28e52621ae5beec59d62fa4292c6a41f942b2f240b43f606595f0f072f43 not found: ID does not exist" containerID="fcbe28e52621ae5beec59d62fa4292c6a41f942b2f240b43f606595f0f072f43" Nov 02 14:58:08.663524 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.663475 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcbe28e52621ae5beec59d62fa4292c6a41f942b2f240b43f606595f0f072f43"} err="failed to get container status \"fcbe28e52621ae5beec59d62fa4292c6a41f942b2f240b43f606595f0f072f43\": rpc error: code = NotFound desc = could not find container \"fcbe28e52621ae5beec59d62fa4292c6a41f942b2f240b43f606595f0f072f43\": container with ID starting with fcbe28e52621ae5beec59d62fa4292c6a41f942b2f240b43f606595f0f072f43 not found: ID does not exist" Nov 02 14:58:08.663524 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.663501 4866 scope.go:122] "RemoveContainer" containerID="1b749e5b8039908924427ff360b6404711128e8112455440d1b6aaf14b398619" Nov 02 14:58:08.672233 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.672212 4866 scope.go:122] "RemoveContainer" containerID="1b749e5b8039908924427ff360b6404711128e8112455440d1b6aaf14b398619" Nov 02 14:58:08.672536 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:58:08.672515 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b749e5b8039908924427ff360b6404711128e8112455440d1b6aaf14b398619\": container with ID starting with 1b749e5b8039908924427ff360b6404711128e8112455440d1b6aaf14b398619 not found: ID does not exist" containerID="1b749e5b8039908924427ff360b6404711128e8112455440d1b6aaf14b398619" Nov 02 14:58:08.672611 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.672549 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b749e5b8039908924427ff360b6404711128e8112455440d1b6aaf14b398619"} err="failed to get container status \"1b749e5b8039908924427ff360b6404711128e8112455440d1b6aaf14b398619\": rpc error: code = NotFound desc = could not find container \"1b749e5b8039908924427ff360b6404711128e8112455440d1b6aaf14b398619\": container with ID starting with 1b749e5b8039908924427ff360b6404711128e8112455440d1b6aaf14b398619 not found: ID does not exist" Nov 02 14:58:08.703327 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.703288 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9"] Nov 02 14:58:08.715152 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.715119 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6018-7f446979f-x7gk9"] Nov 02 14:58:08.748885 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.748783 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s"] Nov 02 14:58:08.752939 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.752906 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6018-6775fc74f6-h829s"] Nov 02 14:58:08.945016 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.944979 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13b35ec4-565c-4b88-867d-638252310831" path="/var/lib/kubelet/pods/13b35ec4-565c-4b88-867d-638252310831/volumes" Nov 02 14:58:08.945372 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:08.945359 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2d2400c-5420-4198-9af1-7e54818595bd" path="/var/lib/kubelet/pods/a2d2400c-5420-4198-9af1-7e54818595bd/volumes" Nov 02 14:58:12.620791 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:12.620600 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv"] Nov 02 14:58:12.621219 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:12.621116 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" podUID="9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" containerName="istio-proxy" containerID="cri-o://2f4d904544ab74401203fe2f1c73154d27349f3ec8973948621dffd596bb00c9" gracePeriod=30 Nov 02 14:58:17.878338 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.878308 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:58:17.883241 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.883216 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-workload-certs\") pod \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " Nov 02 14:58:17.883373 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.883255 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-data\") pod \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " Nov 02 14:58:17.883373 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.883279 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-workload-socket\") pod \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " Nov 02 14:58:17.883373 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.883303 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-podinfo\") pod \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " Nov 02 14:58:17.883373 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.883329 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-token\" (UniqueName: \"kubernetes.io/projected/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-token\") pod \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " Nov 02 14:58:17.883373 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.883351 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-kube-api-access-xcbbb\" (UniqueName: \"kubernetes.io/projected/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-kube-api-access-xcbbb\") pod \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " Nov 02 14:58:17.883643 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.883384 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-credential-socket\") pod \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " Nov 02 14:58:17.883643 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.883532 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-data" pod "9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" (UID: "9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:17.883643 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.883550 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-workload-certs" pod "9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" (UID: "9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:17.883643 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.883605 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-envoy\") pod \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " Nov 02 14:58:17.883871 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.883654 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istiod-ca-cert\") pod \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\" (UID: \"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4\") " Nov 02 14:58:17.883871 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.883664 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-credential-socket" pod "9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" (UID: "9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:17.883871 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.883681 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-workload-socket" pod "9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" (UID: "9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:17.884022 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.883984 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istiod-ca-cert" pod "9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" (UID: "9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:17.884125 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.884109 4866 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-credential-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:58:17.884179 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.884131 4866 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istiod-ca-cert\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:58:17.884179 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.884145 4866 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-workload-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:58:17.884179 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.884160 4866 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-data\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:58:17.884179 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.884172 4866 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-workload-socket\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:58:17.885654 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.885592 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-kube-api-access-xcbbb" pod "9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" (UID: "9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4"). InnerVolumeSpecName "kube-api-access-xcbbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:17.885746 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.885648 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-token" pod "9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" (UID: "9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:17.885921 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.885901 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-podinfo" pod "9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" (UID: "9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:58:17.885921 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.885908 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-envoy" pod "9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" (UID: "9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:17.984732 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.984691 4866 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-envoy\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:58:17.984732 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.984725 4866 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-podinfo\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:58:17.984732 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.984734 4866 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-istio-token\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:58:17.985015 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:17.984745 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcbbb\" (UniqueName: \"kubernetes.io/projected/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4-kube-api-access-xcbbb\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:58:18.707192 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:18.707154 4866 generic.go:356] "Generic (PLEG): container finished" podID="9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" containerID="2f4d904544ab74401203fe2f1c73154d27349f3ec8973948621dffd596bb00c9" exitCode=0 Nov 02 14:58:18.707370 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:18.707187 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" event={"ID":"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4","Type":"ContainerDied","Data":"2f4d904544ab74401203fe2f1c73154d27349f3ec8973948621dffd596bb00c9"} Nov 02 14:58:18.707370 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:18.707210 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" Nov 02 14:58:18.707370 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:18.707250 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv" event={"ID":"9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4","Type":"ContainerDied","Data":"9c63c84656a55931694a56055f708d1d7c6b2532741046174bf659e5bd4cc51d"} Nov 02 14:58:18.707370 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:18.707274 4866 scope.go:122] "RemoveContainer" containerID="2f4d904544ab74401203fe2f1c73154d27349f3ec8973948621dffd596bb00c9" Nov 02 14:58:18.717427 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:18.717403 4866 scope.go:122] "RemoveContainer" containerID="a49e97c5284865666d4eb221665e836a2d09a9cdc8ea21f600a8b7bb33e2837e" Nov 02 14:58:18.725573 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:18.725554 4866 scope.go:122] "RemoveContainer" containerID="2f4d904544ab74401203fe2f1c73154d27349f3ec8973948621dffd596bb00c9" Nov 02 14:58:18.725851 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:58:18.725834 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f4d904544ab74401203fe2f1c73154d27349f3ec8973948621dffd596bb00c9\": container with ID starting with 2f4d904544ab74401203fe2f1c73154d27349f3ec8973948621dffd596bb00c9 not found: ID does not exist" containerID="2f4d904544ab74401203fe2f1c73154d27349f3ec8973948621dffd596bb00c9" Nov 02 14:58:18.725903 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:18.725862 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4d904544ab74401203fe2f1c73154d27349f3ec8973948621dffd596bb00c9"} err="failed to get container status \"2f4d904544ab74401203fe2f1c73154d27349f3ec8973948621dffd596bb00c9\": rpc error: code = NotFound desc = could not find container \"2f4d904544ab74401203fe2f1c73154d27349f3ec8973948621dffd596bb00c9\": container with ID starting with 2f4d904544ab74401203fe2f1c73154d27349f3ec8973948621dffd596bb00c9 not found: ID does not exist" Nov 02 14:58:18.725903 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:18.725879 4866 scope.go:122] "RemoveContainer" containerID="a49e97c5284865666d4eb221665e836a2d09a9cdc8ea21f600a8b7bb33e2837e" Nov 02 14:58:18.726129 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:58:18.726110 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49e97c5284865666d4eb221665e836a2d09a9cdc8ea21f600a8b7bb33e2837e\": container with ID starting with a49e97c5284865666d4eb221665e836a2d09a9cdc8ea21f600a8b7bb33e2837e not found: ID does not exist" containerID="a49e97c5284865666d4eb221665e836a2d09a9cdc8ea21f600a8b7bb33e2837e" Nov 02 14:58:18.726171 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:18.726136 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49e97c5284865666d4eb221665e836a2d09a9cdc8ea21f600a8b7bb33e2837e"} err="failed to get container status \"a49e97c5284865666d4eb221665e836a2d09a9cdc8ea21f600a8b7bb33e2837e\": rpc error: code = NotFound desc = could not find container \"a49e97c5284865666d4eb221665e836a2d09a9cdc8ea21f600a8b7bb33e2837e\": container with ID starting with a49e97c5284865666d4eb221665e836a2d09a9cdc8ea21f600a8b7bb33e2837e not found: ID does not exist" Nov 02 14:58:18.760777 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:18.760736 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv"] Nov 02 14:58:18.772085 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:18.772050 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-be6018-openshift-default-6959f6cfbf-clcpv"] Nov 02 14:58:18.945157 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:18.945123 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" path="/var/lib/kubelet/pods/9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4/volumes" Nov 02 14:58:26.699448 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:26.699407 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-2btkh"] Nov 02 14:58:28.573824 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.573787 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-849d6b95cc-pc2n9"] Nov 02 14:58:28.574398 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.574375 4866 cpu_manager.go:414] "containerMap: removing container" podUID="a2d2400c-5420-4198-9af1-7e54818595bd" containerName="payload-pre-processing" Nov 02 14:58:28.574448 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.574400 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2d2400c-5420-4198-9af1-7e54818595bd" containerName="payload-pre-processing" Nov 02 14:58:28.574448 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.574430 4866 cpu_manager.go:414] "containerMap: removing container" podUID="9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" containerName="istio-proxy" Nov 02 14:58:28.574448 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.574439 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" containerName="istio-proxy" Nov 02 14:58:28.574556 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.574453 4866 cpu_manager.go:414] "containerMap: removing container" podUID="09be7298-cfa0-4d2c-8535-40ff55ec6eba" containerName="maas-api" Nov 02 14:58:28.574556 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.574460 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09be7298-cfa0-4d2c-8535-40ff55ec6eba" containerName="maas-api" Nov 02 14:58:28.574556 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.574474 4866 cpu_manager.go:414] "containerMap: removing container" podUID="9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" containerName="istio-proxy" Nov 02 14:58:28.574556 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.574481 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" containerName="istio-proxy" Nov 02 14:58:28.574556 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.574509 4866 cpu_manager.go:414] "containerMap: removing container" podUID="13b35ec4-565c-4b88-867d-638252310831" containerName="payload-processing" Nov 02 14:58:28.574556 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.574517 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b35ec4-565c-4b88-867d-638252310831" containerName="payload-processing" Nov 02 14:58:28.574556 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.574534 4866 cpu_manager.go:414] "containerMap: removing container" podUID="343fe571-5a17-4108-947b-c393031b192d" containerName="main" Nov 02 14:58:28.574556 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.574540 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="343fe571-5a17-4108-947b-c393031b192d" containerName="main" Nov 02 14:58:28.574826 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.574606 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="13b35ec4-565c-4b88-867d-638252310831" containerName="payload-processing" Nov 02 14:58:28.574826 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.574617 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="343fe571-5a17-4108-947b-c393031b192d" containerName="main" Nov 02 14:58:28.574826 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.574625 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2d2400c-5420-4198-9af1-7e54818595bd" containerName="payload-pre-processing" Nov 02 14:58:28.574826 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.574633 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" containerName="istio-proxy" Nov 02 14:58:28.574826 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.574639 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="09be7298-cfa0-4d2c-8535-40ff55ec6eba" containerName="maas-api" Nov 02 14:58:28.574826 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.574645 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c8b92c0-1cbe-4ed4-bc3f-fde5545a25f4" containerName="istio-proxy" Nov 02 14:58:28.577645 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.577629 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-849d6b95cc-pc2n9" Nov 02 14:58:28.597487 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.597414 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-849d6b95cc-pc2n9"] Nov 02 14:58:28.675500 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.675454 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321a383f-e07d-412b-8aae-78214bb1235b-plugins-config-volume\") pod \"payload-processing-849d6b95cc-pc2n9\" (UID: \"321a383f-e07d-412b-8aae-78214bb1235b\") " pod="openshift-ingress/payload-processing-849d6b95cc-pc2n9" Nov 02 14:58:28.675681 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.675516 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8pmc\" (UniqueName: \"kubernetes.io/projected/321a383f-e07d-412b-8aae-78214bb1235b-kube-api-access-t8pmc\") pod \"payload-processing-849d6b95cc-pc2n9\" (UID: \"321a383f-e07d-412b-8aae-78214bb1235b\") " pod="openshift-ingress/payload-processing-849d6b95cc-pc2n9" Nov 02 14:58:28.776915 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.776870 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321a383f-e07d-412b-8aae-78214bb1235b-plugins-config-volume\") pod \"payload-processing-849d6b95cc-pc2n9\" (UID: \"321a383f-e07d-412b-8aae-78214bb1235b\") " pod="openshift-ingress/payload-processing-849d6b95cc-pc2n9" Nov 02 14:58:28.777156 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.776928 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8pmc\" (UniqueName: \"kubernetes.io/projected/321a383f-e07d-412b-8aae-78214bb1235b-kube-api-access-t8pmc\") pod \"payload-processing-849d6b95cc-pc2n9\" (UID: \"321a383f-e07d-412b-8aae-78214bb1235b\") " pod="openshift-ingress/payload-processing-849d6b95cc-pc2n9" Nov 02 14:58:28.777694 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.777665 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321a383f-e07d-412b-8aae-78214bb1235b-plugins-config-volume\") pod \"payload-processing-849d6b95cc-pc2n9\" (UID: \"321a383f-e07d-412b-8aae-78214bb1235b\") " pod="openshift-ingress/payload-processing-849d6b95cc-pc2n9" Nov 02 14:58:28.800922 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.800889 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8pmc\" (UniqueName: \"kubernetes.io/projected/321a383f-e07d-412b-8aae-78214bb1235b-kube-api-access-t8pmc\") pod \"payload-processing-849d6b95cc-pc2n9\" (UID: \"321a383f-e07d-412b-8aae-78214bb1235b\") " pod="openshift-ingress/payload-processing-849d6b95cc-pc2n9" Nov 02 14:58:28.890616 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:28.890533 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-849d6b95cc-pc2n9" Nov 02 14:58:29.189797 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:29.189771 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-849d6b95cc-pc2n9"] Nov 02 14:58:29.193180 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:58:29.193146 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321a383f_e07d_412b_8aae_78214bb1235b.slice/crio-4e2c01a6e78b80e4c0fcaca0802bb9f9f203d90dc162feb24685aaeea2f25817 WatchSource:0}: Error finding container 4e2c01a6e78b80e4c0fcaca0802bb9f9f203d90dc162feb24685aaeea2f25817: Status 404 returned error can't find the container with id 4e2c01a6e78b80e4c0fcaca0802bb9f9f203d90dc162feb24685aaeea2f25817 Nov 02 14:58:29.760405 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:29.760369 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-849d6b95cc-pc2n9" event={"ID":"321a383f-e07d-412b-8aae-78214bb1235b","Type":"ContainerStarted","Data":"c047496315aa63e87c558eeacdf5cfd760c0210858541dc868dee87181e77d13"} Nov 02 14:58:29.760826 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:29.760424 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-849d6b95cc-pc2n9" event={"ID":"321a383f-e07d-412b-8aae-78214bb1235b","Type":"ContainerStarted","Data":"4e2c01a6e78b80e4c0fcaca0802bb9f9f203d90dc162feb24685aaeea2f25817"} Nov 02 14:58:29.760826 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:29.760445 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-849d6b95cc-pc2n9" Nov 02 14:58:29.802119 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:29.802057 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-849d6b95cc-pc2n9" podStartSLOduration=1.802039086 podStartE2EDuration="1.802039086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:29.800494016 +0000 UTC m=+3674.984341350" watchObservedRunningTime="2026-11-02 14:58:29.802039086 +0000 UTC m=+3674.985886426" Nov 02 14:58:31.700441 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:31.700389 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2btkh" podUID="3edd944f-87df-46e3-aeeb-9873603e1178" containerName="payload-pre-processing" containerID="cri-o://9991431be31da33010d1512fe0fd70617a7c0b2d3ac0e865f6c80d93d20eb377" gracePeriod=25 Nov 02 14:58:31.956644 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:31.956574 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2btkh" Nov 02 14:58:32.009259 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:32.009224 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3edd944f-87df-46e3-aeeb-9873603e1178-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edd944f-87df-46e3-aeeb-9873603e1178-plugins-config-volume\") pod \"3edd944f-87df-46e3-aeeb-9873603e1178\" (UID: \"3edd944f-87df-46e3-aeeb-9873603e1178\") " Nov 02 14:58:32.009406 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:32.009324 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edd944f-87df-46e3-aeeb-9873603e1178-kube-api-access-b7zkc\" (UniqueName: \"kubernetes.io/projected/3edd944f-87df-46e3-aeeb-9873603e1178-kube-api-access-b7zkc\") pod \"3edd944f-87df-46e3-aeeb-9873603e1178\" (UID: \"3edd944f-87df-46e3-aeeb-9873603e1178\") " Nov 02 14:58:32.009686 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:32.009658 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3edd944f-87df-46e3-aeeb-9873603e1178-plugins-config-volume" pod "3edd944f-87df-46e3-aeeb-9873603e1178" (UID: "3edd944f-87df-46e3-aeeb-9873603e1178"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:32.011524 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:32.011492 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edd944f-87df-46e3-aeeb-9873603e1178-kube-api-access-b7zkc" pod "3edd944f-87df-46e3-aeeb-9873603e1178" (UID: "3edd944f-87df-46e3-aeeb-9873603e1178"). InnerVolumeSpecName "kube-api-access-b7zkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:32.110722 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:32.110682 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edd944f-87df-46e3-aeeb-9873603e1178-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:58:32.110722 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:32.110715 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7zkc\" (UniqueName: \"kubernetes.io/projected/3edd944f-87df-46e3-aeeb-9873603e1178-kube-api-access-b7zkc\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:58:32.776006 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:32.775968 4866 generic.go:356] "Generic (PLEG): container finished" podID="3edd944f-87df-46e3-aeeb-9873603e1178" containerID="9991431be31da33010d1512fe0fd70617a7c0b2d3ac0e865f6c80d93d20eb377" exitCode=0 Nov 02 14:58:32.776433 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:32.776013 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2btkh" Nov 02 14:58:32.776433 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:32.776055 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2btkh" event={"ID":"3edd944f-87df-46e3-aeeb-9873603e1178","Type":"ContainerDied","Data":"9991431be31da33010d1512fe0fd70617a7c0b2d3ac0e865f6c80d93d20eb377"} Nov 02 14:58:32.776433 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:32.776109 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2btkh" event={"ID":"3edd944f-87df-46e3-aeeb-9873603e1178","Type":"ContainerDied","Data":"e1a098af69fcfaa939a4bbeaf57616f25c526aa97a30304c8794ce92b517c8cd"} Nov 02 14:58:32.776433 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:32.776130 4866 scope.go:122] "RemoveContainer" containerID="9991431be31da33010d1512fe0fd70617a7c0b2d3ac0e865f6c80d93d20eb377" Nov 02 14:58:32.785032 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:32.785013 4866 scope.go:122] "RemoveContainer" containerID="9991431be31da33010d1512fe0fd70617a7c0b2d3ac0e865f6c80d93d20eb377" Nov 02 14:58:32.785290 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:58:32.785267 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9991431be31da33010d1512fe0fd70617a7c0b2d3ac0e865f6c80d93d20eb377\": container with ID starting with 9991431be31da33010d1512fe0fd70617a7c0b2d3ac0e865f6c80d93d20eb377 not found: ID does not exist" containerID="9991431be31da33010d1512fe0fd70617a7c0b2d3ac0e865f6c80d93d20eb377" Nov 02 14:58:32.785341 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:32.785299 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9991431be31da33010d1512fe0fd70617a7c0b2d3ac0e865f6c80d93d20eb377"} err="failed to get container status \"9991431be31da33010d1512fe0fd70617a7c0b2d3ac0e865f6c80d93d20eb377\": rpc error: code = NotFound desc = could not find container \"9991431be31da33010d1512fe0fd70617a7c0b2d3ac0e865f6c80d93d20eb377\": container with ID starting with 9991431be31da33010d1512fe0fd70617a7c0b2d3ac0e865f6c80d93d20eb377 not found: ID does not exist" Nov 02 14:58:32.826063 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:32.826026 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-2btkh"] Nov 02 14:58:32.830674 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:32.830645 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-2btkh"] Nov 02 14:58:32.944992 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:32.944938 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3edd944f-87df-46e3-aeeb-9873603e1178" path="/var/lib/kubelet/pods/3edd944f-87df-46e3-aeeb-9873603e1178/volumes" Nov 02 14:58:36.009904 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:58:36.009866 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:58:40.769182 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:40.769149 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-849d6b95cc-pc2n9" Nov 02 14:58:40.841367 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:40.841329 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rx5hk"] Nov 02 14:58:42.774657 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:42.774623 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-8f7f4697c-89t4w"] Nov 02 14:58:42.775224 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:42.775198 4866 cpu_manager.go:414] "containerMap: removing container" podUID="3edd944f-87df-46e3-aeeb-9873603e1178" containerName="payload-pre-processing" Nov 02 14:58:42.775401 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:42.775379 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3edd944f-87df-46e3-aeeb-9873603e1178" containerName="payload-pre-processing" Nov 02 14:58:42.775551 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:42.775537 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="3edd944f-87df-46e3-aeeb-9873603e1178" containerName="payload-pre-processing" Nov 02 14:58:42.779773 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:42.779752 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-8f7f4697c-89t4w" Nov 02 14:58:42.798714 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:42.798680 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-8f7f4697c-89t4w"] Nov 02 14:58:42.918440 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:42.918403 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7ltv\" (UniqueName: \"kubernetes.io/projected/dc87147a-63b6-4b08-b45a-e90793367a36-kube-api-access-w7ltv\") pod \"payload-processing-8f7f4697c-89t4w\" (UID: \"dc87147a-63b6-4b08-b45a-e90793367a36\") " pod="openshift-ingress/payload-processing-8f7f4697c-89t4w" Nov 02 14:58:42.918620 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:42.918551 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc87147a-63b6-4b08-b45a-e90793367a36-plugins-config-volume\") pod \"payload-processing-8f7f4697c-89t4w\" (UID: \"dc87147a-63b6-4b08-b45a-e90793367a36\") " pod="openshift-ingress/payload-processing-8f7f4697c-89t4w" Nov 02 14:58:42.961443 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:42.961411 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-bbc4b5f5d-l54ph"] Nov 02 14:58:42.965683 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:42.965663 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-bbc4b5f5d-l54ph" Nov 02 14:58:42.985013 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:42.984982 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-bbc4b5f5d-l54ph"] Nov 02 14:58:43.019400 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.019354 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc87147a-63b6-4b08-b45a-e90793367a36-plugins-config-volume\") pod \"payload-processing-8f7f4697c-89t4w\" (UID: \"dc87147a-63b6-4b08-b45a-e90793367a36\") " pod="openshift-ingress/payload-processing-8f7f4697c-89t4w" Nov 02 14:58:43.019609 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.019450 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlz72\" (UniqueName: \"kubernetes.io/projected/3ce257ea-3be4-412a-92b5-cc93696643c0-kube-api-access-hlz72\") pod \"payload-pre-processing-bbc4b5f5d-l54ph\" (UID: \"3ce257ea-3be4-412a-92b5-cc93696643c0\") " pod="openshift-ingress/payload-pre-processing-bbc4b5f5d-l54ph" Nov 02 14:58:43.019694 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.019649 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7ltv\" (UniqueName: \"kubernetes.io/projected/dc87147a-63b6-4b08-b45a-e90793367a36-kube-api-access-w7ltv\") pod \"payload-processing-8f7f4697c-89t4w\" (UID: \"dc87147a-63b6-4b08-b45a-e90793367a36\") " pod="openshift-ingress/payload-processing-8f7f4697c-89t4w" Nov 02 14:58:43.019760 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.019716 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce257ea-3be4-412a-92b5-cc93696643c0-plugins-config-volume\") pod \"payload-pre-processing-bbc4b5f5d-l54ph\" (UID: \"3ce257ea-3be4-412a-92b5-cc93696643c0\") " pod="openshift-ingress/payload-pre-processing-bbc4b5f5d-l54ph" Nov 02 14:58:43.020337 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.020311 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc87147a-63b6-4b08-b45a-e90793367a36-plugins-config-volume\") pod \"payload-processing-8f7f4697c-89t4w\" (UID: \"dc87147a-63b6-4b08-b45a-e90793367a36\") " pod="openshift-ingress/payload-processing-8f7f4697c-89t4w" Nov 02 14:58:43.043616 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.043542 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7ltv\" (UniqueName: \"kubernetes.io/projected/dc87147a-63b6-4b08-b45a-e90793367a36-kube-api-access-w7ltv\") pod \"payload-processing-8f7f4697c-89t4w\" (UID: \"dc87147a-63b6-4b08-b45a-e90793367a36\") " pod="openshift-ingress/payload-processing-8f7f4697c-89t4w" Nov 02 14:58:43.092197 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.092154 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-8f7f4697c-89t4w" Nov 02 14:58:43.120646 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.120606 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlz72\" (UniqueName: \"kubernetes.io/projected/3ce257ea-3be4-412a-92b5-cc93696643c0-kube-api-access-hlz72\") pod \"payload-pre-processing-bbc4b5f5d-l54ph\" (UID: \"3ce257ea-3be4-412a-92b5-cc93696643c0\") " pod="openshift-ingress/payload-pre-processing-bbc4b5f5d-l54ph" Nov 02 14:58:43.120830 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.120691 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce257ea-3be4-412a-92b5-cc93696643c0-plugins-config-volume\") pod \"payload-pre-processing-bbc4b5f5d-l54ph\" (UID: \"3ce257ea-3be4-412a-92b5-cc93696643c0\") " pod="openshift-ingress/payload-pre-processing-bbc4b5f5d-l54ph" Nov 02 14:58:43.121473 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.121450 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce257ea-3be4-412a-92b5-cc93696643c0-plugins-config-volume\") pod \"payload-pre-processing-bbc4b5f5d-l54ph\" (UID: \"3ce257ea-3be4-412a-92b5-cc93696643c0\") " pod="openshift-ingress/payload-pre-processing-bbc4b5f5d-l54ph" Nov 02 14:58:43.145107 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.145080 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlz72\" (UniqueName: \"kubernetes.io/projected/3ce257ea-3be4-412a-92b5-cc93696643c0-kube-api-access-hlz72\") pod \"payload-pre-processing-bbc4b5f5d-l54ph\" (UID: \"3ce257ea-3be4-412a-92b5-cc93696643c0\") " pod="openshift-ingress/payload-pre-processing-bbc4b5f5d-l54ph" Nov 02 14:58:43.277688 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.277660 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-bbc4b5f5d-l54ph" Nov 02 14:58:43.393139 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.393111 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-8f7f4697c-89t4w"] Nov 02 14:58:43.395569 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:58:43.395537 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc87147a_63b6_4b08_b45a_e90793367a36.slice/crio-ef877aac858e3c89383cf84ca69852af98c2cf809dc7b9c20d4effea124ccddb WatchSource:0}: Error finding container ef877aac858e3c89383cf84ca69852af98c2cf809dc7b9c20d4effea124ccddb: Status 404 returned error can't find the container with id ef877aac858e3c89383cf84ca69852af98c2cf809dc7b9c20d4effea124ccddb Nov 02 14:58:43.584534 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.584289 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-bbc4b5f5d-l54ph"] Nov 02 14:58:43.587337 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:58:43.587307 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce257ea_3be4_412a_92b5_cc93696643c0.slice/crio-91676a68420d0a620b070526950a0370af5671164969181fecca38fa2dffb2c5 WatchSource:0}: Error finding container 91676a68420d0a620b070526950a0370af5671164969181fecca38fa2dffb2c5: Status 404 returned error can't find the container with id 91676a68420d0a620b070526950a0370af5671164969181fecca38fa2dffb2c5 Nov 02 14:58:43.833384 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.833342 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-bbc4b5f5d-l54ph" event={"ID":"3ce257ea-3be4-412a-92b5-cc93696643c0","Type":"ContainerStarted","Data":"ad3909dfb9c98585513e2f5b847e996b7b2251d1593d5ab0574d5ce84112f4e4"} Nov 02 14:58:43.833841 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.833411 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-bbc4b5f5d-l54ph" event={"ID":"3ce257ea-3be4-412a-92b5-cc93696643c0","Type":"ContainerStarted","Data":"91676a68420d0a620b070526950a0370af5671164969181fecca38fa2dffb2c5"} Nov 02 14:58:43.833841 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.833531 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-bbc4b5f5d-l54ph" Nov 02 14:58:43.835052 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.834992 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-8f7f4697c-89t4w" event={"ID":"dc87147a-63b6-4b08-b45a-e90793367a36","Type":"ContainerStarted","Data":"9b73d0e4596dd956590c31bf6007e5e60e4160e689c7825a75543efa6a4e7b37"} Nov 02 14:58:43.835052 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.835040 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-8f7f4697c-89t4w" event={"ID":"dc87147a-63b6-4b08-b45a-e90793367a36","Type":"ContainerStarted","Data":"ef877aac858e3c89383cf84ca69852af98c2cf809dc7b9c20d4effea124ccddb"} Nov 02 14:58:43.835177 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.835088 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-8f7f4697c-89t4w" Nov 02 14:58:43.874841 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.874782 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-bbc4b5f5d-l54ph" podStartSLOduration=1.874764539 podStartE2EDuration="1.874764539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:43.872775244 +0000 UTC m=+3689.056622581" watchObservedRunningTime="2026-11-02 14:58:43.874764539 +0000 UTC m=+3689.058611877" Nov 02 14:58:43.912968 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:43.912895 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-8f7f4697c-89t4w" podStartSLOduration=1.912881609 podStartE2EDuration="1.912881609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:43.912601821 +0000 UTC m=+3689.096449158" watchObservedRunningTime="2026-11-02 14:58:43.912881609 +0000 UTC m=+3689.096728946" Nov 02 14:58:45.842304 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:45.842258 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx5hk" podUID="ce41c5ab-a5da-44b5-bf3e-f045c70f55ce" containerName="payload-processing" containerID="cri-o://da39a05ebdf68e22e0121a37b53694cc73f15b44fc8eabe10aede7b080381abe" gracePeriod=25 Nov 02 14:58:46.096194 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:46.096134 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx5hk" Nov 02 14:58:46.255669 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:46.255634 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce41c5ab-a5da-44b5-bf3e-f045c70f55ce-kube-api-access-lhsj6\" (UniqueName: \"kubernetes.io/projected/ce41c5ab-a5da-44b5-bf3e-f045c70f55ce-kube-api-access-lhsj6\") pod \"ce41c5ab-a5da-44b5-bf3e-f045c70f55ce\" (UID: \"ce41c5ab-a5da-44b5-bf3e-f045c70f55ce\") " Nov 02 14:58:46.255669 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:46.255669 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce41c5ab-a5da-44b5-bf3e-f045c70f55ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce41c5ab-a5da-44b5-bf3e-f045c70f55ce-plugins-config-volume\") pod \"ce41c5ab-a5da-44b5-bf3e-f045c70f55ce\" (UID: \"ce41c5ab-a5da-44b5-bf3e-f045c70f55ce\") " Nov 02 14:58:46.256095 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:46.256074 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce41c5ab-a5da-44b5-bf3e-f045c70f55ce-plugins-config-volume" pod "ce41c5ab-a5da-44b5-bf3e-f045c70f55ce" (UID: "ce41c5ab-a5da-44b5-bf3e-f045c70f55ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:46.257821 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:46.257793 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce41c5ab-a5da-44b5-bf3e-f045c70f55ce-kube-api-access-lhsj6" pod "ce41c5ab-a5da-44b5-bf3e-f045c70f55ce" (UID: "ce41c5ab-a5da-44b5-bf3e-f045c70f55ce"). InnerVolumeSpecName "kube-api-access-lhsj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:46.356521 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:46.356437 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhsj6\" (UniqueName: \"kubernetes.io/projected/ce41c5ab-a5da-44b5-bf3e-f045c70f55ce-kube-api-access-lhsj6\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:58:46.356521 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:46.356470 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce41c5ab-a5da-44b5-bf3e-f045c70f55ce-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:58:46.850820 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:46.850784 4866 generic.go:356] "Generic (PLEG): container finished" podID="ce41c5ab-a5da-44b5-bf3e-f045c70f55ce" containerID="da39a05ebdf68e22e0121a37b53694cc73f15b44fc8eabe10aede7b080381abe" exitCode=0 Nov 02 14:58:46.851336 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:46.850833 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx5hk" Nov 02 14:58:46.851336 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:46.850868 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx5hk" event={"ID":"ce41c5ab-a5da-44b5-bf3e-f045c70f55ce","Type":"ContainerDied","Data":"da39a05ebdf68e22e0121a37b53694cc73f15b44fc8eabe10aede7b080381abe"} Nov 02 14:58:46.851336 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:46.850928 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx5hk" event={"ID":"ce41c5ab-a5da-44b5-bf3e-f045c70f55ce","Type":"ContainerDied","Data":"0da2f80dc0c12786e964995393e1eeb64d232800091ed76a974b1badfe8958a2"} Nov 02 14:58:46.851336 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:46.850990 4866 scope.go:122] "RemoveContainer" containerID="da39a05ebdf68e22e0121a37b53694cc73f15b44fc8eabe10aede7b080381abe" Nov 02 14:58:46.860302 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:46.860282 4866 scope.go:122] "RemoveContainer" containerID="da39a05ebdf68e22e0121a37b53694cc73f15b44fc8eabe10aede7b080381abe" Nov 02 14:58:46.860586 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:58:46.860565 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da39a05ebdf68e22e0121a37b53694cc73f15b44fc8eabe10aede7b080381abe\": container with ID starting with da39a05ebdf68e22e0121a37b53694cc73f15b44fc8eabe10aede7b080381abe not found: ID does not exist" containerID="da39a05ebdf68e22e0121a37b53694cc73f15b44fc8eabe10aede7b080381abe" Nov 02 14:58:46.860651 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:46.860599 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da39a05ebdf68e22e0121a37b53694cc73f15b44fc8eabe10aede7b080381abe"} err="failed to get container status \"da39a05ebdf68e22e0121a37b53694cc73f15b44fc8eabe10aede7b080381abe\": rpc error: code = NotFound desc = could not find container \"da39a05ebdf68e22e0121a37b53694cc73f15b44fc8eabe10aede7b080381abe\": container with ID starting with da39a05ebdf68e22e0121a37b53694cc73f15b44fc8eabe10aede7b080381abe not found: ID does not exist" Nov 02 14:58:46.903447 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:46.903414 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rx5hk"] Nov 02 14:58:46.914937 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:46.914908 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rx5hk"] Nov 02 14:58:46.944781 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:46.944750 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce41c5ab-a5da-44b5-bf3e-f045c70f55ce" path="/var/lib/kubelet/pods/ce41c5ab-a5da-44b5-bf3e-f045c70f55ce/volumes" Nov 02 14:58:54.842754 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:54.842721 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-8f7f4697c-89t4w" Nov 02 14:58:54.843319 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:54.842789 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-bbc4b5f5d-l54ph" Nov 02 14:58:54.915260 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:54.915225 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-849d6b95cc-pc2n9"] Nov 02 14:58:58.432735 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.432697 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n"] Nov 02 14:58:58.433509 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.433483 4866 cpu_manager.go:414] "containerMap: removing container" podUID="ce41c5ab-a5da-44b5-bf3e-f045c70f55ce" containerName="payload-processing" Nov 02 14:58:58.433617 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.433510 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce41c5ab-a5da-44b5-bf3e-f045c70f55ce" containerName="payload-processing" Nov 02 14:58:58.433675 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.433651 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce41c5ab-a5da-44b5-bf3e-f045c70f55ce" containerName="payload-processing" Nov 02 14:58:58.436672 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.436649 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" Nov 02 14:58:58.451157 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.451134 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 02 14:58:58.462689 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.462662 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-mdllg\"" Nov 02 14:58:58.462689 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.462671 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 14:58:58.462907 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.462795 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 14:58:58.462907 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.462808 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 14:58:58.462907 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.462816 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 14:58:58.465687 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.465658 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n"] Nov 02 14:58:58.574989 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.574955 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-n5n4n\" (UID: \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" Nov 02 14:58:58.575201 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.575004 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw7df\" (UniqueName: \"kubernetes.io/projected/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-kube-api-access-vw7df\") pod \"payload-processing-external-model-5bcd7997-n5n4n\" (UID: \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" Nov 02 14:58:58.575277 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.575197 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-n5n4n\" (UID: \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" Nov 02 14:58:58.575331 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.575307 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-n5n4n\" (UID: \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" Nov 02 14:58:58.676532 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.676496 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-n5n4n\" (UID: \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" Nov 02 14:58:58.676738 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.676545 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw7df\" (UniqueName: \"kubernetes.io/projected/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-kube-api-access-vw7df\") pod \"payload-processing-external-model-5bcd7997-n5n4n\" (UID: \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" Nov 02 14:58:58.676738 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.676610 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-n5n4n\" (UID: \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" Nov 02 14:58:58.676738 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.676644 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-n5n4n\" (UID: \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" Nov 02 14:58:58.677330 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.677300 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-n5n4n\" (UID: \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" Nov 02 14:58:58.677444 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.677406 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-n5n4n\" (UID: \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" Nov 02 14:58:58.679060 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.679041 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-n5n4n\" (UID: \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" Nov 02 14:58:58.701629 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.701604 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw7df\" (UniqueName: \"kubernetes.io/projected/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-kube-api-access-vw7df\") pod \"payload-processing-external-model-5bcd7997-n5n4n\" (UID: \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" Nov 02 14:58:58.749517 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:58.749484 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" Nov 02 14:58:59.053524 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:59.053481 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n"] Nov 02 14:58:59.055511 ip-10-0-134-99 kubenswrapper[4866]: W1102 14:58:59.055482 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf511ef39_2e38_4f88_b15e_ceb5ecc65dfe.slice/crio-1214e36951d571004b10a62fb45b09b409cda0430bd794179b171a5ae02346b2 WatchSource:0}: Error finding container 1214e36951d571004b10a62fb45b09b409cda0430bd794179b171a5ae02346b2: Status 404 returned error can't find the container with id 1214e36951d571004b10a62fb45b09b409cda0430bd794179b171a5ae02346b2 Nov 02 14:58:59.914826 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:59.914790 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" event={"ID":"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe","Type":"ContainerStarted","Data":"cef0564786912f7abae7c544bbbc50e41a4ea8b9fe673b7d03cd6be89c540deb"} Nov 02 14:58:59.915224 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:59.914842 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" event={"ID":"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe","Type":"ContainerStarted","Data":"1214e36951d571004b10a62fb45b09b409cda0430bd794179b171a5ae02346b2"} Nov 02 14:58:59.915224 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:59.914873 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" Nov 02 14:58:59.916129 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:59.916095 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-849d6b95cc-pc2n9" podUID="321a383f-e07d-412b-8aae-78214bb1235b" containerName="payload-processing" containerID="cri-o://c047496315aa63e87c558eeacdf5cfd760c0210858541dc868dee87181e77d13" gracePeriod=25 Nov 02 14:58:59.959622 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:58:59.959562 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" podStartSLOduration=1.959546139 podStartE2EDuration="1.959546139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:59.958364293 +0000 UTC m=+3705.142211631" watchObservedRunningTime="2026-11-02 14:58:59.959546139 +0000 UTC m=+3705.143393477" Nov 02 14:59:00.192864 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:59:00.192829 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-849d6b95cc-pc2n9" Nov 02 14:59:00.291969 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:59:00.291911 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/321a383f-e07d-412b-8aae-78214bb1235b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321a383f-e07d-412b-8aae-78214bb1235b-plugins-config-volume\") pod \"321a383f-e07d-412b-8aae-78214bb1235b\" (UID: \"321a383f-e07d-412b-8aae-78214bb1235b\") " Nov 02 14:59:00.292179 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:59:00.292016 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321a383f-e07d-412b-8aae-78214bb1235b-kube-api-access-t8pmc\" (UniqueName: \"kubernetes.io/projected/321a383f-e07d-412b-8aae-78214bb1235b-kube-api-access-t8pmc\") pod \"321a383f-e07d-412b-8aae-78214bb1235b\" (UID: \"321a383f-e07d-412b-8aae-78214bb1235b\") " Nov 02 14:59:00.292413 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:59:00.292380 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321a383f-e07d-412b-8aae-78214bb1235b-plugins-config-volume" pod "321a383f-e07d-412b-8aae-78214bb1235b" (UID: "321a383f-e07d-412b-8aae-78214bb1235b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:59:00.294526 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:59:00.294499 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321a383f-e07d-412b-8aae-78214bb1235b-kube-api-access-t8pmc" pod "321a383f-e07d-412b-8aae-78214bb1235b" (UID: "321a383f-e07d-412b-8aae-78214bb1235b"). InnerVolumeSpecName "kube-api-access-t8pmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:59:00.399968 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:59:00.393470 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321a383f-e07d-412b-8aae-78214bb1235b-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:59:00.399968 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:59:00.393514 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8pmc\" (UniqueName: \"kubernetes.io/projected/321a383f-e07d-412b-8aae-78214bb1235b-kube-api-access-t8pmc\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 14:59:00.923291 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:59:00.923255 4866 generic.go:356] "Generic (PLEG): container finished" podID="321a383f-e07d-412b-8aae-78214bb1235b" containerID="c047496315aa63e87c558eeacdf5cfd760c0210858541dc868dee87181e77d13" exitCode=0 Nov 02 14:59:00.923772 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:59:00.923345 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-849d6b95cc-pc2n9" Nov 02 14:59:00.923772 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:59:00.923355 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-849d6b95cc-pc2n9" event={"ID":"321a383f-e07d-412b-8aae-78214bb1235b","Type":"ContainerDied","Data":"c047496315aa63e87c558eeacdf5cfd760c0210858541dc868dee87181e77d13"} Nov 02 14:59:00.923772 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:59:00.923463 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-849d6b95cc-pc2n9" event={"ID":"321a383f-e07d-412b-8aae-78214bb1235b","Type":"ContainerDied","Data":"4e2c01a6e78b80e4c0fcaca0802bb9f9f203d90dc162feb24685aaeea2f25817"} Nov 02 14:59:00.923772 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:59:00.923498 4866 scope.go:122] "RemoveContainer" containerID="c047496315aa63e87c558eeacdf5cfd760c0210858541dc868dee87181e77d13" Nov 02 14:59:00.933909 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:59:00.933879 4866 scope.go:122] "RemoveContainer" containerID="c047496315aa63e87c558eeacdf5cfd760c0210858541dc868dee87181e77d13" Nov 02 14:59:00.934279 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:59:00.934246 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c047496315aa63e87c558eeacdf5cfd760c0210858541dc868dee87181e77d13\": container with ID starting with c047496315aa63e87c558eeacdf5cfd760c0210858541dc868dee87181e77d13 not found: ID does not exist" containerID="c047496315aa63e87c558eeacdf5cfd760c0210858541dc868dee87181e77d13" Nov 02 14:59:00.934458 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:59:00.934425 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c047496315aa63e87c558eeacdf5cfd760c0210858541dc868dee87181e77d13"} err="failed to get container status \"c047496315aa63e87c558eeacdf5cfd760c0210858541dc868dee87181e77d13\": rpc error: code = NotFound desc = could not find container \"c047496315aa63e87c558eeacdf5cfd760c0210858541dc868dee87181e77d13\": container with ID starting with c047496315aa63e87c558eeacdf5cfd760c0210858541dc868dee87181e77d13 not found: ID does not exist" Nov 02 14:59:00.975616 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:59:00.975569 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-849d6b95cc-pc2n9"] Nov 02 14:59:00.987135 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:59:00.987096 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-849d6b95cc-pc2n9"] Nov 02 14:59:02.949637 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:59:02.949601 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="321a383f-e07d-412b-8aae-78214bb1235b" path="/var/lib/kubelet/pods/321a383f-e07d-412b-8aae-78214bb1235b/volumes" Nov 02 14:59:06.010164 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:59:06.010125 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 14:59:10.927462 ip-10-0-134-99 kubenswrapper[4866]: I1102 14:59:10.927429 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" Nov 02 14:59:36.009646 ip-10-0-134-99 kubenswrapper[4866]: E1102 14:59:36.009557 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:00:06.009715 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:00:06.009681 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:00:36.008371 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:00:36.008331 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:00:46.162755 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:46.162710 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n"] Nov 02 15:00:51.163534 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.163484 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" podUID="f511ef39-2e38-4f88-b15e-ceb5ecc65dfe" containerName="payload-processing" containerID="cri-o://cef0564786912f7abae7c544bbbc50e41a4ea8b9fe673b7d03cd6be89c540deb" gracePeriod=25 Nov 02 15:00:51.415735 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.415661 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" Nov 02 15:00:51.489226 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.489186 4866 generic.go:356] "Generic (PLEG): container finished" podID="f511ef39-2e38-4f88-b15e-ceb5ecc65dfe" containerID="cef0564786912f7abae7c544bbbc50e41a4ea8b9fe673b7d03cd6be89c540deb" exitCode=0 Nov 02 15:00:51.489226 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.489227 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" Nov 02 15:00:51.489496 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.489256 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" event={"ID":"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe","Type":"ContainerDied","Data":"cef0564786912f7abae7c544bbbc50e41a4ea8b9fe673b7d03cd6be89c540deb"} Nov 02 15:00:51.489496 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.489311 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n" event={"ID":"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe","Type":"ContainerDied","Data":"1214e36951d571004b10a62fb45b09b409cda0430bd794179b171a5ae02346b2"} Nov 02 15:00:51.489496 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.489336 4866 scope.go:122] "RemoveContainer" containerID="cef0564786912f7abae7c544bbbc50e41a4ea8b9fe673b7d03cd6be89c540deb" Nov 02 15:00:51.499024 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.498933 4866 scope.go:122] "RemoveContainer" containerID="cef0564786912f7abae7c544bbbc50e41a4ea8b9fe673b7d03cd6be89c540deb" Nov 02 15:00:51.499335 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:00:51.499307 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef0564786912f7abae7c544bbbc50e41a4ea8b9fe673b7d03cd6be89c540deb\": container with ID starting with cef0564786912f7abae7c544bbbc50e41a4ea8b9fe673b7d03cd6be89c540deb not found: ID does not exist" containerID="cef0564786912f7abae7c544bbbc50e41a4ea8b9fe673b7d03cd6be89c540deb" Nov 02 15:00:51.499412 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.499340 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef0564786912f7abae7c544bbbc50e41a4ea8b9fe673b7d03cd6be89c540deb"} err="failed to get container status \"cef0564786912f7abae7c544bbbc50e41a4ea8b9fe673b7d03cd6be89c540deb\": rpc error: code = NotFound desc = could not find container \"cef0564786912f7abae7c544bbbc50e41a4ea8b9fe673b7d03cd6be89c540deb\": container with ID starting with cef0564786912f7abae7c544bbbc50e41a4ea8b9fe673b7d03cd6be89c540deb not found: ID does not exist" Nov 02 15:00:51.507413 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.507392 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-plugins-config-volume\") pod \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\" (UID: \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\") " Nov 02 15:00:51.507507 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.507496 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-routing-overlay-volume\") pod \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\" (UID: \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\") " Nov 02 15:00:51.507558 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.507547 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-kube-api-access-vw7df\" (UniqueName: \"kubernetes.io/projected/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-kube-api-access-vw7df\") pod \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\" (UID: \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\") " Nov 02 15:00:51.507600 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.507577 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-provider-credentials\") pod \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\" (UID: \"f511ef39-2e38-4f88-b15e-ceb5ecc65dfe\") " Nov 02 15:00:51.507793 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.507772 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-plugins-config-volume" pod "f511ef39-2e38-4f88-b15e-ceb5ecc65dfe" (UID: "f511ef39-2e38-4f88-b15e-ceb5ecc65dfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:00:51.507899 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.507875 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-routing-overlay-volume" pod "f511ef39-2e38-4f88-b15e-ceb5ecc65dfe" (UID: "f511ef39-2e38-4f88-b15e-ceb5ecc65dfe"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:00:51.509634 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.509615 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-provider-credentials" pod "f511ef39-2e38-4f88-b15e-ceb5ecc65dfe" (UID: "f511ef39-2e38-4f88-b15e-ceb5ecc65dfe"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:00:51.509727 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.509710 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-kube-api-access-vw7df" pod "f511ef39-2e38-4f88-b15e-ceb5ecc65dfe" (UID: "f511ef39-2e38-4f88-b15e-ceb5ecc65dfe"). InnerVolumeSpecName "kube-api-access-vw7df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:00:51.608417 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.608381 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw7df\" (UniqueName: \"kubernetes.io/projected/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-kube-api-access-vw7df\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:00:51.608417 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.608413 4866 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-provider-credentials\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:00:51.608417 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.608426 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:00:51.608712 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.608440 4866 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe-routing-overlay-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:00:51.836156 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.836124 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n"] Nov 02 15:00:51.847525 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:51.847491 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-n5n4n"] Nov 02 15:00:52.944766 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:00:52.944730 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f511ef39-2e38-4f88-b15e-ceb5ecc65dfe" path="/var/lib/kubelet/pods/f511ef39-2e38-4f88-b15e-ceb5ecc65dfe/volumes" Nov 02 15:01:06.008421 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:01:06.008341 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:01:36.008762 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:01:36.008722 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:02:06.014217 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:02:06.014164 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:02:15.701774 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:02:15.701740 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 15:02:15.721886 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:02:15.721859 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 15:02:36.009146 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:02:36.009059 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:03:06.008715 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:03:06.008675 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:03:36.008469 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:03:36.008428 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:04:03.966209 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:03.966114 4866 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-8q5hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:04:03.966827 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:03.966775 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-8q5hv" podUID="1fbf701c-a1aa-4031-8112-4e69e00e103b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:04:06.009651 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:04:06.009612 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:04:36.009261 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:04:36.009224 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:04:40.656866 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:40.656819 4866 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" podUID="fcd82cc88301759f1e08e2b014a50651" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:04:55.267272 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.267237 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s"] Nov 02 15:04:55.267849 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.267684 4866 cpu_manager.go:414] "containerMap: removing container" podUID="f511ef39-2e38-4f88-b15e-ceb5ecc65dfe" containerName="payload-processing" Nov 02 15:04:55.267849 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.267696 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f511ef39-2e38-4f88-b15e-ceb5ecc65dfe" containerName="payload-processing" Nov 02 15:04:55.267849 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.267722 4866 cpu_manager.go:414] "containerMap: removing container" podUID="321a383f-e07d-412b-8aae-78214bb1235b" containerName="payload-processing" Nov 02 15:04:55.267849 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.267731 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321a383f-e07d-412b-8aae-78214bb1235b" containerName="payload-processing" Nov 02 15:04:55.267849 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.267796 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="321a383f-e07d-412b-8aae-78214bb1235b" containerName="payload-processing" Nov 02 15:04:55.267849 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.267817 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="f511ef39-2e38-4f88-b15e-ceb5ecc65dfe" containerName="payload-processing" Nov 02 15:04:55.271172 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.271150 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" Nov 02 15:04:55.285189 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.285157 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw75s\" (UniqueName: \"kubernetes.io/projected/f6dc1633-658d-4a1c-bfec-92b285b08b15-kube-api-access-lw75s\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s\" (UID: \"f6dc1633-658d-4a1c-bfec-92b285b08b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" Nov 02 15:04:55.285337 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.285215 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6dc1633-658d-4a1c-bfec-92b285b08b15-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s\" (UID: \"f6dc1633-658d-4a1c-bfec-92b285b08b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" Nov 02 15:04:55.285337 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.285301 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dc1633-658d-4a1c-bfec-92b285b08b15-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s\" (UID: \"f6dc1633-658d-4a1c-bfec-92b285b08b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" Nov 02 15:04:55.285436 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.285346 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6dc1633-658d-4a1c-bfec-92b285b08b15-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s\" (UID: \"f6dc1633-658d-4a1c-bfec-92b285b08b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" Nov 02 15:04:55.289514 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.289486 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-4bc210-serving-cert\"" Nov 02 15:04:55.302827 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.302792 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s"] Nov 02 15:04:55.386295 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.386261 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6dc1633-658d-4a1c-bfec-92b285b08b15-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s\" (UID: \"f6dc1633-658d-4a1c-bfec-92b285b08b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" Nov 02 15:04:55.386470 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.386306 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dc1633-658d-4a1c-bfec-92b285b08b15-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s\" (UID: \"f6dc1633-658d-4a1c-bfec-92b285b08b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" Nov 02 15:04:55.386470 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.386336 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6dc1633-658d-4a1c-bfec-92b285b08b15-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s\" (UID: \"f6dc1633-658d-4a1c-bfec-92b285b08b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" Nov 02 15:04:55.386470 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.386438 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw75s\" (UniqueName: \"kubernetes.io/projected/f6dc1633-658d-4a1c-bfec-92b285b08b15-kube-api-access-lw75s\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s\" (UID: \"f6dc1633-658d-4a1c-bfec-92b285b08b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" Nov 02 15:04:55.387160 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.387135 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6dc1633-658d-4a1c-bfec-92b285b08b15-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s\" (UID: \"f6dc1633-658d-4a1c-bfec-92b285b08b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" Nov 02 15:04:55.388741 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.388709 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dc1633-658d-4a1c-bfec-92b285b08b15-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s\" (UID: \"f6dc1633-658d-4a1c-bfec-92b285b08b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" Nov 02 15:04:55.388872 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.388813 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6dc1633-658d-4a1c-bfec-92b285b08b15-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s\" (UID: \"f6dc1633-658d-4a1c-bfec-92b285b08b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" Nov 02 15:04:55.409393 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.409360 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw75s\" (UniqueName: \"kubernetes.io/projected/f6dc1633-658d-4a1c-bfec-92b285b08b15-kube-api-access-lw75s\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s\" (UID: \"f6dc1633-658d-4a1c-bfec-92b285b08b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" Nov 02 15:04:55.530066 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.529978 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st"] Nov 02 15:04:55.534176 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.534155 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" Nov 02 15:04:55.552698 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.552657 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-4bc210\"" Nov 02 15:04:55.553020 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.553000 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-14d73c7f-dockercfg-n2mdh\"" Nov 02 15:04:55.554595 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.554572 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st"] Nov 02 15:04:55.584745 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.584703 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" Nov 02 15:04:55.588209 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.588182 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tspb\" (UniqueName: \"kubernetes.io/projected/818e13cc-77d1-448b-a537-a8152ef116b8-kube-api-access-8tspb\") pod \"payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st\" (UID: \"818e13cc-77d1-448b-a537-a8152ef116b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" Nov 02 15:04:55.588995 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.588968 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818e13cc-77d1-448b-a537-a8152ef116b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st\" (UID: \"818e13cc-77d1-448b-a537-a8152ef116b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" Nov 02 15:04:55.615679 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.613712 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2"] Nov 02 15:04:55.618995 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.618906 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" Nov 02 15:04:55.633508 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.633475 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2"] Nov 02 15:04:55.689708 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.689677 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818e13cc-77d1-448b-a537-a8152ef116b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st\" (UID: \"818e13cc-77d1-448b-a537-a8152ef116b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" Nov 02 15:04:55.689888 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.689744 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/282f0691-0348-4a9e-a2f3-c07b481a9489-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2\" (UID: \"282f0691-0348-4a9e-a2f3-c07b481a9489\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" Nov 02 15:04:55.689888 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.689813 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tspb\" (UniqueName: \"kubernetes.io/projected/818e13cc-77d1-448b-a537-a8152ef116b8-kube-api-access-8tspb\") pod \"payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st\" (UID: \"818e13cc-77d1-448b-a537-a8152ef116b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" Nov 02 15:04:55.689888 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.689851 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4vs9\" (UniqueName: \"kubernetes.io/projected/282f0691-0348-4a9e-a2f3-c07b481a9489-kube-api-access-q4vs9\") pod \"payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2\" (UID: \"282f0691-0348-4a9e-a2f3-c07b481a9489\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" Nov 02 15:04:55.690527 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.690500 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818e13cc-77d1-448b-a537-a8152ef116b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st\" (UID: \"818e13cc-77d1-448b-a537-a8152ef116b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" Nov 02 15:04:55.713082 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.713047 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tspb\" (UniqueName: \"kubernetes.io/projected/818e13cc-77d1-448b-a537-a8152ef116b8-kube-api-access-8tspb\") pod \"payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st\" (UID: \"818e13cc-77d1-448b-a537-a8152ef116b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" Nov 02 15:04:55.791335 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.791249 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/282f0691-0348-4a9e-a2f3-c07b481a9489-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2\" (UID: \"282f0691-0348-4a9e-a2f3-c07b481a9489\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" Nov 02 15:04:55.791335 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.791305 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4vs9\" (UniqueName: \"kubernetes.io/projected/282f0691-0348-4a9e-a2f3-c07b481a9489-kube-api-access-q4vs9\") pod \"payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2\" (UID: \"282f0691-0348-4a9e-a2f3-c07b481a9489\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" Nov 02 15:04:55.792002 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.791942 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/282f0691-0348-4a9e-a2f3-c07b481a9489-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2\" (UID: \"282f0691-0348-4a9e-a2f3-c07b481a9489\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" Nov 02 15:04:55.814340 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.814313 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4vs9\" (UniqueName: \"kubernetes.io/projected/282f0691-0348-4a9e-a2f3-c07b481a9489-kube-api-access-q4vs9\") pod \"payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2\" (UID: \"282f0691-0348-4a9e-a2f3-c07b481a9489\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" Nov 02 15:04:55.847763 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.847738 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" Nov 02 15:04:55.913767 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.913732 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s"] Nov 02 15:04:55.918299 ip-10-0-134-99 kubenswrapper[4866]: W1102 15:04:55.918270 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6dc1633_658d_4a1c_bfec_92b285b08b15.slice/crio-478b2ec0c8b00288b8d4c17eaba6af3fffab6b989045d189592e75adc81e7d9c WatchSource:0}: Error finding container 478b2ec0c8b00288b8d4c17eaba6af3fffab6b989045d189592e75adc81e7d9c: Status 404 returned error can't find the container with id 478b2ec0c8b00288b8d4c17eaba6af3fffab6b989045d189592e75adc81e7d9c Nov 02 15:04:55.919592 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.919575 4866 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:04:55.934819 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:55.934638 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" Nov 02 15:04:56.145942 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:56.145854 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st"] Nov 02 15:04:56.148157 ip-10-0-134-99 kubenswrapper[4866]: W1102 15:04:56.148131 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818e13cc_77d1_448b_a537_a8152ef116b8.slice/crio-1500f39a2e5fdb4427c6f601152e843f10d7b47f0b45807105b5b7b46b61d7f8 WatchSource:0}: Error finding container 1500f39a2e5fdb4427c6f601152e843f10d7b47f0b45807105b5b7b46b61d7f8: Status 404 returned error can't find the container with id 1500f39a2e5fdb4427c6f601152e843f10d7b47f0b45807105b5b7b46b61d7f8 Nov 02 15:04:56.233702 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:56.233594 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2"] Nov 02 15:04:56.236780 ip-10-0-134-99 kubenswrapper[4866]: W1102 15:04:56.236750 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282f0691_0348_4a9e_a2f3_c07b481a9489.slice/crio-b920755c721b05e2c7ec8e88cb45d8087421cc9dc5ee52be4d2f0dd29ef556c0 WatchSource:0}: Error finding container b920755c721b05e2c7ec8e88cb45d8087421cc9dc5ee52be4d2f0dd29ef556c0: Status 404 returned error can't find the container with id b920755c721b05e2c7ec8e88cb45d8087421cc9dc5ee52be4d2f0dd29ef556c0 Nov 02 15:04:56.752768 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:56.752735 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" event={"ID":"f6dc1633-658d-4a1c-bfec-92b285b08b15","Type":"ContainerStarted","Data":"a02c06d615cc62f8423cb51aa6b5876dbbac8483d8a8308e455ee7589578d8a1"} Nov 02 15:04:56.753240 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:56.752825 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" event={"ID":"f6dc1633-658d-4a1c-bfec-92b285b08b15","Type":"ContainerStarted","Data":"478b2ec0c8b00288b8d4c17eaba6af3fffab6b989045d189592e75adc81e7d9c"} Nov 02 15:04:56.753240 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:56.752998 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" Nov 02 15:04:56.755483 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:56.755458 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" event={"ID":"282f0691-0348-4a9e-a2f3-c07b481a9489","Type":"ContainerStarted","Data":"87b08d2394ba6e88bb98f18b1b0fd140fc98f8ee36b70121a0d91659efdb2adf"} Nov 02 15:04:56.755630 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:56.755499 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" event={"ID":"282f0691-0348-4a9e-a2f3-c07b481a9489","Type":"ContainerStarted","Data":"b920755c721b05e2c7ec8e88cb45d8087421cc9dc5ee52be4d2f0dd29ef556c0"} Nov 02 15:04:56.755630 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:56.755522 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" Nov 02 15:04:56.757033 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:56.757010 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" event={"ID":"818e13cc-77d1-448b-a537-a8152ef116b8","Type":"ContainerStarted","Data":"2e09ae564e74107de9e177ce10648cb1f9b60896f300be05c6ba8742fd722a20"} Nov 02 15:04:56.757145 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:56.757046 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" event={"ID":"818e13cc-77d1-448b-a537-a8152ef116b8","Type":"ContainerStarted","Data":"1500f39a2e5fdb4427c6f601152e843f10d7b47f0b45807105b5b7b46b61d7f8"} Nov 02 15:04:56.757184 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:56.757138 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" Nov 02 15:04:56.795840 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:56.795792 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" podStartSLOduration=1.295784351 podStartE2EDuration="1.795778375s" totalImagesPullingTime="499.994024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:56.794079292 +0000 UTC m=+4061.977926630" watchObservedRunningTime="2026-11-02 15:04:56.795778375 +0000 UTC m=+4061.979625712" Nov 02 15:04:56.835205 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:56.835161 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" podStartSLOduration=1.835148134 podStartE2EDuration="1.835148134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:56.834161283 +0000 UTC m=+4062.018008620" watchObservedRunningTime="2026-11-02 15:04:56.835148134 +0000 UTC m=+4062.018995472" Nov 02 15:04:56.874446 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:04:56.874394 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" podStartSLOduration=1.874380467 podStartE2EDuration="1.874380467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:56.873673138 +0000 UTC m=+4062.057520475" watchObservedRunningTime="2026-11-02 15:04:56.874380467 +0000 UTC m=+4062.058227803" Nov 02 15:05:02.767324 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:02.767293 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" Nov 02 15:05:06.009294 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:05:06.009255 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:05:07.764458 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:07.764428 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" Nov 02 15:05:07.764860 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:07.764496 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" Nov 02 15:05:11.146834 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.146796 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps"] Nov 02 15:05:11.151738 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.151711 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.180254 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.180220 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-4bc210\"/\"simulator-single-main-4bc210-kserve-self-signed-certs\"" Nov 02 15:05:11.180254 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.180236 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-4bc210\"/\"kube-root-ca.crt\"" Nov 02 15:05:11.180508 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.180390 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-4bc210\"/\"openshift-service-ca.crt\"" Nov 02 15:05:11.181913 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.181888 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps"] Nov 02 15:05:11.248535 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.248487 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-tmp-dir\") pod \"simulator-single-main-4bc210-kserve-85cf9db768-wxgps\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.248718 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.248559 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l5jx\" (UniqueName: \"kubernetes.io/projected/823daade-a067-4761-98f9-4e6543429e4e-kube-api-access-2l5jx\") pod \"simulator-single-main-4bc210-kserve-85cf9db768-wxgps\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.248718 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.248664 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-home\") pod \"simulator-single-main-4bc210-kserve-85cf9db768-wxgps\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.248718 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.248687 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823daade-a067-4761-98f9-4e6543429e4e-tls-certs\") pod \"simulator-single-main-4bc210-kserve-85cf9db768-wxgps\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.248718 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.248708 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-model-cache\") pod \"simulator-single-main-4bc210-kserve-85cf9db768-wxgps\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.248857 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.248824 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-dshm\") pod \"simulator-single-main-4bc210-kserve-85cf9db768-wxgps\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.265182 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.265156 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-4bc210\"/\"default-dockercfg-2xv5j\"" Nov 02 15:05:11.349544 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.349503 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-tmp-dir\") pod \"simulator-single-main-4bc210-kserve-85cf9db768-wxgps\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.349708 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.349553 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l5jx\" (UniqueName: \"kubernetes.io/projected/823daade-a067-4761-98f9-4e6543429e4e-kube-api-access-2l5jx\") pod \"simulator-single-main-4bc210-kserve-85cf9db768-wxgps\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.349708 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.349606 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-home\") pod \"simulator-single-main-4bc210-kserve-85cf9db768-wxgps\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.349708 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.349622 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823daade-a067-4761-98f9-4e6543429e4e-tls-certs\") pod \"simulator-single-main-4bc210-kserve-85cf9db768-wxgps\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.349708 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.349639 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-model-cache\") pod \"simulator-single-main-4bc210-kserve-85cf9db768-wxgps\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.349708 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.349676 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-dshm\") pod \"simulator-single-main-4bc210-kserve-85cf9db768-wxgps\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.350035 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.350008 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-tmp-dir\") pod \"simulator-single-main-4bc210-kserve-85cf9db768-wxgps\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.350112 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.350043 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-home\") pod \"simulator-single-main-4bc210-kserve-85cf9db768-wxgps\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.350184 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.350162 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-model-cache\") pod \"simulator-single-main-4bc210-kserve-85cf9db768-wxgps\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.352017 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.351993 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-dshm\") pod \"simulator-single-main-4bc210-kserve-85cf9db768-wxgps\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.352228 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.352209 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823daade-a067-4761-98f9-4e6543429e4e-tls-certs\") pod \"simulator-single-main-4bc210-kserve-85cf9db768-wxgps\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.386261 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.386233 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l5jx\" (UniqueName: \"kubernetes.io/projected/823daade-a067-4761-98f9-4e6543429e4e-kube-api-access-2l5jx\") pod \"simulator-single-main-4bc210-kserve-85cf9db768-wxgps\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.463581 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.463548 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:11.761793 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.761764 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps"] Nov 02 15:05:11.763709 ip-10-0-134-99 kubenswrapper[4866]: W1102 15:05:11.763681 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod823daade_a067_4761_98f9_4e6543429e4e.slice/crio-fd793b46ef8ed772fb76d14d6baeb54668a835a8693555d63172f5ed4d5a347b WatchSource:0}: Error finding container fd793b46ef8ed772fb76d14d6baeb54668a835a8693555d63172f5ed4d5a347b: Status 404 returned error can't find the container with id fd793b46ef8ed772fb76d14d6baeb54668a835a8693555d63172f5ed4d5a347b Nov 02 15:05:11.827102 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.827069 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" event={"ID":"823daade-a067-4761-98f9-4e6543429e4e","Type":"ContainerStarted","Data":"aa11a3e45c8d0eafbd5491b001ef04691bb45f0232ea8b8afafeb772fd4dcf1e"} Nov 02 15:05:11.827227 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.827124 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" event={"ID":"823daade-a067-4761-98f9-4e6543429e4e","Type":"ContainerStarted","Data":"fd793b46ef8ed772fb76d14d6baeb54668a835a8693555d63172f5ed4d5a347b"} Nov 02 15:05:11.868434 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:11.868383 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" podStartSLOduration=0.868368182 podStartE2EDuration="868.368182ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:11.867624921 +0000 UTC m=+4077.051472257" watchObservedRunningTime="2026-11-02 15:05:11.868368182 +0000 UTC m=+4077.052215519" Nov 02 15:05:12.464134 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:12.464096 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:21.464658 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:21.464602 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:21.478336 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:21.478309 4866 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:21.892265 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:21.892178 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:05:36.009084 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:05:36.008977 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:05:59.119991 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:05:59.119954 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps"] Nov 02 15:06:01.228305 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:01.228260 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2"] Nov 02 15:06:01.262609 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:01.262572 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st"] Nov 02 15:06:04.327607 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:04.327562 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s"] Nov 02 15:06:04.328090 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:04.327791 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" podUID="f6dc1633-658d-4a1c-bfec-92b285b08b15" containerName="maas-api" containerID="cri-o://a02c06d615cc62f8423cb51aa6b5876dbbac8483d8a8308e455ee7589578d8a1" gracePeriod=30 Nov 02 15:06:04.792063 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:04.792035 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" Nov 02 15:06:04.871335 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:04.871251 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6dc1633-658d-4a1c-bfec-92b285b08b15-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6dc1633-658d-4a1c-bfec-92b285b08b15-maas-api-tls\") pod \"f6dc1633-658d-4a1c-bfec-92b285b08b15\" (UID: \"f6dc1633-658d-4a1c-bfec-92b285b08b15\") " Nov 02 15:06:04.871482 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:04.871356 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6dc1633-658d-4a1c-bfec-92b285b08b15-kube-api-access-lw75s\" (UniqueName: \"kubernetes.io/projected/f6dc1633-658d-4a1c-bfec-92b285b08b15-kube-api-access-lw75s\") pod \"f6dc1633-658d-4a1c-bfec-92b285b08b15\" (UID: \"f6dc1633-658d-4a1c-bfec-92b285b08b15\") " Nov 02 15:06:04.871482 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:04.871382 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6dc1633-658d-4a1c-bfec-92b285b08b15-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dc1633-658d-4a1c-bfec-92b285b08b15-metrics-certs\") pod \"f6dc1633-658d-4a1c-bfec-92b285b08b15\" (UID: \"f6dc1633-658d-4a1c-bfec-92b285b08b15\") " Nov 02 15:06:04.871556 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:04.871504 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6dc1633-658d-4a1c-bfec-92b285b08b15-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6dc1633-658d-4a1c-bfec-92b285b08b15-openshift-service-ca\") pod \"f6dc1633-658d-4a1c-bfec-92b285b08b15\" (UID: \"f6dc1633-658d-4a1c-bfec-92b285b08b15\") " Nov 02 15:06:04.871922 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:04.871890 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6dc1633-658d-4a1c-bfec-92b285b08b15-openshift-service-ca" pod "f6dc1633-658d-4a1c-bfec-92b285b08b15" (UID: "f6dc1633-658d-4a1c-bfec-92b285b08b15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:06:04.873551 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:04.873521 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6dc1633-658d-4a1c-bfec-92b285b08b15-metrics-certs" pod "f6dc1633-658d-4a1c-bfec-92b285b08b15" (UID: "f6dc1633-658d-4a1c-bfec-92b285b08b15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:06:04.873646 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:04.873550 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6dc1633-658d-4a1c-bfec-92b285b08b15-maas-api-tls" pod "f6dc1633-658d-4a1c-bfec-92b285b08b15" (UID: "f6dc1633-658d-4a1c-bfec-92b285b08b15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:06:04.873646 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:04.873551 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6dc1633-658d-4a1c-bfec-92b285b08b15-kube-api-access-lw75s" pod "f6dc1633-658d-4a1c-bfec-92b285b08b15" (UID: "f6dc1633-658d-4a1c-bfec-92b285b08b15"). InnerVolumeSpecName "kube-api-access-lw75s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:06:04.972660 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:04.972630 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6dc1633-658d-4a1c-bfec-92b285b08b15-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:06:04.972660 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:04.972657 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6dc1633-658d-4a1c-bfec-92b285b08b15-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:06:04.972660 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:04.972667 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw75s\" (UniqueName: \"kubernetes.io/projected/f6dc1633-658d-4a1c-bfec-92b285b08b15-kube-api-access-lw75s\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:06:04.972900 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:04.972676 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dc1633-658d-4a1c-bfec-92b285b08b15-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:06:05.087169 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:05.087132 4866 generic.go:356] "Generic (PLEG): container finished" podID="f6dc1633-658d-4a1c-bfec-92b285b08b15" containerID="a02c06d615cc62f8423cb51aa6b5876dbbac8483d8a8308e455ee7589578d8a1" exitCode=0 Nov 02 15:06:05.087338 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:05.087179 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" Nov 02 15:06:05.087338 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:05.087216 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" event={"ID":"f6dc1633-658d-4a1c-bfec-92b285b08b15","Type":"ContainerDied","Data":"a02c06d615cc62f8423cb51aa6b5876dbbac8483d8a8308e455ee7589578d8a1"} Nov 02 15:06:05.087338 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:05.087268 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s" event={"ID":"f6dc1633-658d-4a1c-bfec-92b285b08b15","Type":"ContainerDied","Data":"478b2ec0c8b00288b8d4c17eaba6af3fffab6b989045d189592e75adc81e7d9c"} Nov 02 15:06:05.087338 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:05.087292 4866 scope.go:122] "RemoveContainer" containerID="a02c06d615cc62f8423cb51aa6b5876dbbac8483d8a8308e455ee7589578d8a1" Nov 02 15:06:05.096132 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:05.096109 4866 scope.go:122] "RemoveContainer" containerID="a02c06d615cc62f8423cb51aa6b5876dbbac8483d8a8308e455ee7589578d8a1" Nov 02 15:06:05.096372 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:06:05.096349 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02c06d615cc62f8423cb51aa6b5876dbbac8483d8a8308e455ee7589578d8a1\": container with ID starting with a02c06d615cc62f8423cb51aa6b5876dbbac8483d8a8308e455ee7589578d8a1 not found: ID does not exist" containerID="a02c06d615cc62f8423cb51aa6b5876dbbac8483d8a8308e455ee7589578d8a1" Nov 02 15:06:05.096438 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:05.096382 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02c06d615cc62f8423cb51aa6b5876dbbac8483d8a8308e455ee7589578d8a1"} err="failed to get container status \"a02c06d615cc62f8423cb51aa6b5876dbbac8483d8a8308e455ee7589578d8a1\": rpc error: code = NotFound desc = could not find container \"a02c06d615cc62f8423cb51aa6b5876dbbac8483d8a8308e455ee7589578d8a1\": container with ID starting with a02c06d615cc62f8423cb51aa6b5876dbbac8483d8a8308e455ee7589578d8a1 not found: ID does not exist" Nov 02 15:06:05.137836 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:05.137752 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s"] Nov 02 15:06:05.142289 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:05.142267 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-vls9s"] Nov 02 15:06:06.009345 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:06:06.009309 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:06:06.229051 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:06.228987 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" podUID="282f0691-0348-4a9e-a2f3-c07b481a9489" containerName="payload-processing" containerID="cri-o://87b08d2394ba6e88bb98f18b1b0fd140fc98f8ee36b70121a0d91659efdb2adf" gracePeriod=25 Nov 02 15:06:06.263453 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:06.263368 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" podUID="818e13cc-77d1-448b-a537-a8152ef116b8" containerName="payload-pre-processing" containerID="cri-o://2e09ae564e74107de9e177ce10648cb1f9b60896f300be05c6ba8742fd722a20" gracePeriod=25 Nov 02 15:06:06.945545 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:06.945511 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6dc1633-658d-4a1c-bfec-92b285b08b15" path="/var/lib/kubelet/pods/f6dc1633-658d-4a1c-bfec-92b285b08b15/volumes" Nov 02 15:06:07.763205 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:07.763167 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" podUID="818e13cc-77d1-448b-a537-a8152ef116b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:06:07.763664 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:07.763182 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" podUID="282f0691-0348-4a9e-a2f3-c07b481a9489" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:06:14.139688 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.139648 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" podUID="823daade-a067-4761-98f9-4e6543429e4e" containerName="main" containerID="cri-o://aa11a3e45c8d0eafbd5491b001ef04691bb45f0232ea8b8afafeb772fd4dcf1e" gracePeriod=45 Nov 02 15:06:14.390257 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.390189 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:06:14.467223 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.467192 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823daade-a067-4761-98f9-4e6543429e4e-kube-api-access-2l5jx\" (UniqueName: \"kubernetes.io/projected/823daade-a067-4761-98f9-4e6543429e4e-kube-api-access-2l5jx\") pod \"823daade-a067-4761-98f9-4e6543429e4e\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " Nov 02 15:06:14.467417 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.467234 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-dshm\") pod \"823daade-a067-4761-98f9-4e6543429e4e\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " Nov 02 15:06:14.467417 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.467284 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-home\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-home\") pod \"823daade-a067-4761-98f9-4e6543429e4e\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " Nov 02 15:06:14.467417 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.467318 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-model-cache\") pod \"823daade-a067-4761-98f9-4e6543429e4e\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " Nov 02 15:06:14.467417 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.467383 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-tmp-dir\") pod \"823daade-a067-4761-98f9-4e6543429e4e\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " Nov 02 15:06:14.467417 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.467410 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823daade-a067-4761-98f9-4e6543429e4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/823daade-a067-4761-98f9-4e6543429e4e-tls-certs\") pod \"823daade-a067-4761-98f9-4e6543429e4e\" (UID: \"823daade-a067-4761-98f9-4e6543429e4e\") " Nov 02 15:06:14.467769 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.467735 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-model-cache" pod "823daade-a067-4761-98f9-4e6543429e4e" (UID: "823daade-a067-4761-98f9-4e6543429e4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:06:14.467911 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.467817 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-home" pod "823daade-a067-4761-98f9-4e6543429e4e" (UID: "823daade-a067-4761-98f9-4e6543429e4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:06:14.468015 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.467910 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-tmp-dir" pod "823daade-a067-4761-98f9-4e6543429e4e" (UID: "823daade-a067-4761-98f9-4e6543429e4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:06:14.469889 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.469863 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823daade-a067-4761-98f9-4e6543429e4e-tls-certs" pod "823daade-a067-4761-98f9-4e6543429e4e" (UID: "823daade-a067-4761-98f9-4e6543429e4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:06:14.470060 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.469931 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-dshm" pod "823daade-a067-4761-98f9-4e6543429e4e" (UID: "823daade-a067-4761-98f9-4e6543429e4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:06:14.470145 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.470070 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823daade-a067-4761-98f9-4e6543429e4e-kube-api-access-2l5jx" pod "823daade-a067-4761-98f9-4e6543429e4e" (UID: "823daade-a067-4761-98f9-4e6543429e4e"). InnerVolumeSpecName "kube-api-access-2l5jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:06:14.568555 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.568521 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l5jx\" (UniqueName: \"kubernetes.io/projected/823daade-a067-4761-98f9-4e6543429e4e-kube-api-access-2l5jx\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:06:14.568555 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.568548 4866 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-dshm\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:06:14.568555 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.568557 4866 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-home\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:06:14.568798 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.568566 4866 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-model-cache\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:06:14.568798 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.568574 4866 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823daade-a067-4761-98f9-4e6543429e4e-tmp-dir\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:06:14.568798 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:14.568584 4866 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823daade-a067-4761-98f9-4e6543429e4e-tls-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:06:15.143388 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:15.143354 4866 generic.go:356] "Generic (PLEG): container finished" podID="823daade-a067-4761-98f9-4e6543429e4e" containerID="aa11a3e45c8d0eafbd5491b001ef04691bb45f0232ea8b8afafeb772fd4dcf1e" exitCode=0 Nov 02 15:06:15.143887 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:15.143398 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" Nov 02 15:06:15.143887 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:15.143433 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" event={"ID":"823daade-a067-4761-98f9-4e6543429e4e","Type":"ContainerDied","Data":"aa11a3e45c8d0eafbd5491b001ef04691bb45f0232ea8b8afafeb772fd4dcf1e"} Nov 02 15:06:15.143887 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:15.143474 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps" event={"ID":"823daade-a067-4761-98f9-4e6543429e4e","Type":"ContainerDied","Data":"fd793b46ef8ed772fb76d14d6baeb54668a835a8693555d63172f5ed4d5a347b"} Nov 02 15:06:15.143887 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:15.143490 4866 scope.go:122] "RemoveContainer" containerID="aa11a3e45c8d0eafbd5491b001ef04691bb45f0232ea8b8afafeb772fd4dcf1e" Nov 02 15:06:15.152101 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:15.152056 4866 scope.go:122] "RemoveContainer" containerID="aa11a3e45c8d0eafbd5491b001ef04691bb45f0232ea8b8afafeb772fd4dcf1e" Nov 02 15:06:15.152330 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:06:15.152310 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa11a3e45c8d0eafbd5491b001ef04691bb45f0232ea8b8afafeb772fd4dcf1e\": container with ID starting with aa11a3e45c8d0eafbd5491b001ef04691bb45f0232ea8b8afafeb772fd4dcf1e not found: ID does not exist" containerID="aa11a3e45c8d0eafbd5491b001ef04691bb45f0232ea8b8afafeb772fd4dcf1e" Nov 02 15:06:15.152371 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:15.152338 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa11a3e45c8d0eafbd5491b001ef04691bb45f0232ea8b8afafeb772fd4dcf1e"} err="failed to get container status \"aa11a3e45c8d0eafbd5491b001ef04691bb45f0232ea8b8afafeb772fd4dcf1e\": rpc error: code = NotFound desc = could not find container \"aa11a3e45c8d0eafbd5491b001ef04691bb45f0232ea8b8afafeb772fd4dcf1e\": container with ID starting with aa11a3e45c8d0eafbd5491b001ef04691bb45f0232ea8b8afafeb772fd4dcf1e not found: ID does not exist" Nov 02 15:06:15.193432 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:15.193397 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps"] Nov 02 15:06:15.204783 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:15.204712 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-4bc210/simulator-single-main-4bc210-kserve-85cf9db768-wxgps"] Nov 02 15:06:16.945475 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:16.945444 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="823daade-a067-4761-98f9-4e6543429e4e" path="/var/lib/kubelet/pods/823daade-a067-4761-98f9-4e6543429e4e/volumes" Nov 02 15:06:17.763261 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:17.763222 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" podUID="282f0691-0348-4a9e-a2f3-c07b481a9489" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:06:17.763261 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:17.763239 4866 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" podUID="818e13cc-77d1-448b-a537-a8152ef116b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:06:26.527573 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:26.527545 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" Nov 02 15:06:26.553461 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:26.553433 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" Nov 02 15:06:26.579152 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:26.579114 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818e13cc-77d1-448b-a537-a8152ef116b8-kube-api-access-8tspb\" (UniqueName: \"kubernetes.io/projected/818e13cc-77d1-448b-a537-a8152ef116b8-kube-api-access-8tspb\") pod \"818e13cc-77d1-448b-a537-a8152ef116b8\" (UID: \"818e13cc-77d1-448b-a537-a8152ef116b8\") " Nov 02 15:06:26.579152 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:26.579156 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/282f0691-0348-4a9e-a2f3-c07b481a9489-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/282f0691-0348-4a9e-a2f3-c07b481a9489-plugins-config-volume\") pod \"282f0691-0348-4a9e-a2f3-c07b481a9489\" (UID: \"282f0691-0348-4a9e-a2f3-c07b481a9489\") " Nov 02 15:06:26.579396 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:26.579185 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/282f0691-0348-4a9e-a2f3-c07b481a9489-kube-api-access-q4vs9\" (UniqueName: \"kubernetes.io/projected/282f0691-0348-4a9e-a2f3-c07b481a9489-kube-api-access-q4vs9\") pod \"282f0691-0348-4a9e-a2f3-c07b481a9489\" (UID: \"282f0691-0348-4a9e-a2f3-c07b481a9489\") " Nov 02 15:06:26.579396 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:26.579234 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/818e13cc-77d1-448b-a537-a8152ef116b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818e13cc-77d1-448b-a537-a8152ef116b8-plugins-config-volume\") pod \"818e13cc-77d1-448b-a537-a8152ef116b8\" (UID: \"818e13cc-77d1-448b-a537-a8152ef116b8\") " Nov 02 15:06:26.579618 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:26.579583 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/282f0691-0348-4a9e-a2f3-c07b481a9489-plugins-config-volume" pod "282f0691-0348-4a9e-a2f3-c07b481a9489" (UID: "282f0691-0348-4a9e-a2f3-c07b481a9489"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:06:26.579742 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:26.579647 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818e13cc-77d1-448b-a537-a8152ef116b8-plugins-config-volume" pod "818e13cc-77d1-448b-a537-a8152ef116b8" (UID: "818e13cc-77d1-448b-a537-a8152ef116b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:06:26.581418 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:26.581389 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818e13cc-77d1-448b-a537-a8152ef116b8-kube-api-access-8tspb" pod "818e13cc-77d1-448b-a537-a8152ef116b8" (UID: "818e13cc-77d1-448b-a537-a8152ef116b8"). InnerVolumeSpecName "kube-api-access-8tspb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:06:26.581549 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:26.581478 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/282f0691-0348-4a9e-a2f3-c07b481a9489-kube-api-access-q4vs9" pod "282f0691-0348-4a9e-a2f3-c07b481a9489" (UID: "282f0691-0348-4a9e-a2f3-c07b481a9489"). InnerVolumeSpecName "kube-api-access-q4vs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:06:26.680939 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:26.680892 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tspb\" (UniqueName: \"kubernetes.io/projected/818e13cc-77d1-448b-a537-a8152ef116b8-kube-api-access-8tspb\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:06:26.680939 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:26.680928 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/282f0691-0348-4a9e-a2f3-c07b481a9489-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:06:26.680939 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:26.680938 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4vs9\" (UniqueName: \"kubernetes.io/projected/282f0691-0348-4a9e-a2f3-c07b481a9489-kube-api-access-q4vs9\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:06:26.680939 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:26.680959 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818e13cc-77d1-448b-a537-a8152ef116b8-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:06:27.200253 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:27.200220 4866 generic.go:356] "Generic (PLEG): container finished" podID="282f0691-0348-4a9e-a2f3-c07b481a9489" containerID="87b08d2394ba6e88bb98f18b1b0fd140fc98f8ee36b70121a0d91659efdb2adf" exitCode=0 Nov 02 15:06:27.200463 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:27.200268 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" Nov 02 15:06:27.200463 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:27.200294 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" event={"ID":"282f0691-0348-4a9e-a2f3-c07b481a9489","Type":"ContainerDied","Data":"87b08d2394ba6e88bb98f18b1b0fd140fc98f8ee36b70121a0d91659efdb2adf"} Nov 02 15:06:27.200463 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:27.200341 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2" event={"ID":"282f0691-0348-4a9e-a2f3-c07b481a9489","Type":"ContainerDied","Data":"b920755c721b05e2c7ec8e88cb45d8087421cc9dc5ee52be4d2f0dd29ef556c0"} Nov 02 15:06:27.200463 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:27.200372 4866 scope.go:122] "RemoveContainer" containerID="87b08d2394ba6e88bb98f18b1b0fd140fc98f8ee36b70121a0d91659efdb2adf" Nov 02 15:06:27.201639 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:27.201611 4866 generic.go:356] "Generic (PLEG): container finished" podID="818e13cc-77d1-448b-a537-a8152ef116b8" containerID="2e09ae564e74107de9e177ce10648cb1f9b60896f300be05c6ba8742fd722a20" exitCode=0 Nov 02 15:06:27.201774 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:27.201647 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" event={"ID":"818e13cc-77d1-448b-a537-a8152ef116b8","Type":"ContainerDied","Data":"2e09ae564e74107de9e177ce10648cb1f9b60896f300be05c6ba8742fd722a20"} Nov 02 15:06:27.201774 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:27.201672 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" Nov 02 15:06:27.201774 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:27.201695 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st" event={"ID":"818e13cc-77d1-448b-a537-a8152ef116b8","Type":"ContainerDied","Data":"1500f39a2e5fdb4427c6f601152e843f10d7b47f0b45807105b5b7b46b61d7f8"} Nov 02 15:06:27.209357 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:27.209339 4866 scope.go:122] "RemoveContainer" containerID="87b08d2394ba6e88bb98f18b1b0fd140fc98f8ee36b70121a0d91659efdb2adf" Nov 02 15:06:27.209627 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:06:27.209605 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b08d2394ba6e88bb98f18b1b0fd140fc98f8ee36b70121a0d91659efdb2adf\": container with ID starting with 87b08d2394ba6e88bb98f18b1b0fd140fc98f8ee36b70121a0d91659efdb2adf not found: ID does not exist" containerID="87b08d2394ba6e88bb98f18b1b0fd140fc98f8ee36b70121a0d91659efdb2adf" Nov 02 15:06:27.209703 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:27.209636 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b08d2394ba6e88bb98f18b1b0fd140fc98f8ee36b70121a0d91659efdb2adf"} err="failed to get container status \"87b08d2394ba6e88bb98f18b1b0fd140fc98f8ee36b70121a0d91659efdb2adf\": rpc error: code = NotFound desc = could not find container \"87b08d2394ba6e88bb98f18b1b0fd140fc98f8ee36b70121a0d91659efdb2adf\": container with ID starting with 87b08d2394ba6e88bb98f18b1b0fd140fc98f8ee36b70121a0d91659efdb2adf not found: ID does not exist" Nov 02 15:06:27.209703 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:27.209665 4866 scope.go:122] "RemoveContainer" containerID="2e09ae564e74107de9e177ce10648cb1f9b60896f300be05c6ba8742fd722a20" Nov 02 15:06:27.217481 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:27.217460 4866 scope.go:122] "RemoveContainer" containerID="2e09ae564e74107de9e177ce10648cb1f9b60896f300be05c6ba8742fd722a20" Nov 02 15:06:27.217730 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:06:27.217710 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e09ae564e74107de9e177ce10648cb1f9b60896f300be05c6ba8742fd722a20\": container with ID starting with 2e09ae564e74107de9e177ce10648cb1f9b60896f300be05c6ba8742fd722a20 not found: ID does not exist" containerID="2e09ae564e74107de9e177ce10648cb1f9b60896f300be05c6ba8742fd722a20" Nov 02 15:06:27.217790 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:27.217740 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e09ae564e74107de9e177ce10648cb1f9b60896f300be05c6ba8742fd722a20"} err="failed to get container status \"2e09ae564e74107de9e177ce10648cb1f9b60896f300be05c6ba8742fd722a20\": rpc error: code = NotFound desc = could not find container \"2e09ae564e74107de9e177ce10648cb1f9b60896f300be05c6ba8742fd722a20\": container with ID starting with 2e09ae564e74107de9e177ce10648cb1f9b60896f300be05c6ba8742fd722a20 not found: ID does not exist" Nov 02 15:06:27.249668 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:27.249632 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st"] Nov 02 15:06:27.261181 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:27.261157 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7775st"] Nov 02 15:06:27.294711 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:27.294680 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2"] Nov 02 15:06:27.298246 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:27.298220 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cfr2qq2"] Nov 02 15:06:28.945131 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:28.945100 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="282f0691-0348-4a9e-a2f3-c07b481a9489" path="/var/lib/kubelet/pods/282f0691-0348-4a9e-a2f3-c07b481a9489/volumes" Nov 02 15:06:28.945535 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:28.945478 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="818e13cc-77d1-448b-a537-a8152ef116b8" path="/var/lib/kubelet/pods/818e13cc-77d1-448b-a537-a8152ef116b8/volumes" Nov 02 15:06:36.008706 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:06:36.008665 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:06:36.460791 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.460754 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw"] Nov 02 15:06:36.461209 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.461195 4866 cpu_manager.go:414] "containerMap: removing container" podUID="818e13cc-77d1-448b-a537-a8152ef116b8" containerName="payload-pre-processing" Nov 02 15:06:36.461287 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.461210 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818e13cc-77d1-448b-a537-a8152ef116b8" containerName="payload-pre-processing" Nov 02 15:06:36.461287 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.461232 4866 cpu_manager.go:414] "containerMap: removing container" podUID="f6dc1633-658d-4a1c-bfec-92b285b08b15" containerName="maas-api" Nov 02 15:06:36.461287 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.461237 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6dc1633-658d-4a1c-bfec-92b285b08b15" containerName="maas-api" Nov 02 15:06:36.461287 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.461252 4866 cpu_manager.go:414] "containerMap: removing container" podUID="823daade-a067-4761-98f9-4e6543429e4e" containerName="main" Nov 02 15:06:36.461287 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.461257 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823daade-a067-4761-98f9-4e6543429e4e" containerName="main" Nov 02 15:06:36.461287 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.461275 4866 cpu_manager.go:414] "containerMap: removing container" podUID="282f0691-0348-4a9e-a2f3-c07b481a9489" containerName="payload-processing" Nov 02 15:06:36.461287 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.461280 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="282f0691-0348-4a9e-a2f3-c07b481a9489" containerName="payload-processing" Nov 02 15:06:36.461576 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.461345 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="823daade-a067-4761-98f9-4e6543429e4e" containerName="main" Nov 02 15:06:36.461576 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.461357 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="282f0691-0348-4a9e-a2f3-c07b481a9489" containerName="payload-processing" Nov 02 15:06:36.461576 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.461365 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="818e13cc-77d1-448b-a537-a8152ef116b8" containerName="payload-pre-processing" Nov 02 15:06:36.461576 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.461372 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6dc1633-658d-4a1c-bfec-92b285b08b15" containerName="maas-api" Nov 02 15:06:36.466313 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.466296 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" Nov 02 15:06:36.484142 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.484089 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-4bc210-serving-cert\"" Nov 02 15:06:36.498795 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.498758 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw"] Nov 02 15:06:36.575504 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.575470 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcvsg\" (UniqueName: \"kubernetes.io/projected/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-kube-api-access-wcvsg\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw\" (UID: \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" Nov 02 15:06:36.575707 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.575538 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw\" (UID: \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" Nov 02 15:06:36.575707 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.575658 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw\" (UID: \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" Nov 02 15:06:36.575835 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.575783 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw\" (UID: \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" Nov 02 15:06:36.676626 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.676595 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw\" (UID: \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" Nov 02 15:06:36.676811 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.676637 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw\" (UID: \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" Nov 02 15:06:36.676811 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.676696 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw\" (UID: \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" Nov 02 15:06:36.676811 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.676726 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcvsg\" (UniqueName: \"kubernetes.io/projected/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-kube-api-access-wcvsg\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw\" (UID: \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" Nov 02 15:06:36.677395 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.677371 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw\" (UID: \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" Nov 02 15:06:36.679180 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.679156 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw\" (UID: \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" Nov 02 15:06:36.679274 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.679210 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw\" (UID: \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" Nov 02 15:06:36.700662 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.700630 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcvsg\" (UniqueName: \"kubernetes.io/projected/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-kube-api-access-wcvsg\") pod \"maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw\" (UID: \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" Nov 02 15:06:36.778432 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.778349 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" Nov 02 15:06:36.949175 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.949141 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb"] Nov 02 15:06:36.953018 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.952996 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb" Nov 02 15:06:36.967978 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.967935 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-4bc210\"" Nov 02 15:06:36.968110 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.968059 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-14d73c7f-dockercfg-gnsfj\"" Nov 02 15:06:36.974174 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.974148 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb"] Nov 02 15:06:36.996453 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:36.996337 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z"] Nov 02 15:06:37.000401 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.000381 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z" Nov 02 15:06:37.019982 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.019935 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z"] Nov 02 15:06:37.081269 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.081245 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea31d24-c22c-43da-af1a-e45518e760d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb\" (UID: \"3ea31d24-c22c-43da-af1a-e45518e760d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb" Nov 02 15:06:37.081329 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.081282 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3d924e2-760f-4a18-89bf-3d7e0a73086c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z\" (UID: \"e3d924e2-760f-4a18-89bf-3d7e0a73086c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z" Nov 02 15:06:37.081456 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.081429 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bbqv\" (UniqueName: \"kubernetes.io/projected/e3d924e2-760f-4a18-89bf-3d7e0a73086c-kube-api-access-2bbqv\") pod \"payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z\" (UID: \"e3d924e2-760f-4a18-89bf-3d7e0a73086c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z" Nov 02 15:06:37.081559 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.081546 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk98q\" (UniqueName: \"kubernetes.io/projected/3ea31d24-c22c-43da-af1a-e45518e760d8-kube-api-access-mk98q\") pod \"payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb\" (UID: \"3ea31d24-c22c-43da-af1a-e45518e760d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb" Nov 02 15:06:37.081706 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.081686 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw"] Nov 02 15:06:37.083444 ip-10-0-134-99 kubenswrapper[4866]: W1102 15:06:37.083419 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1f25eb_1a57_45b8_9afd_169ad9d6ea0a.slice/crio-f193858818d02cf4d514d172bfba50e6891e5b68e9d0be9f01efe757b42bde35 WatchSource:0}: Error finding container f193858818d02cf4d514d172bfba50e6891e5b68e9d0be9f01efe757b42bde35: Status 404 returned error can't find the container with id f193858818d02cf4d514d172bfba50e6891e5b68e9d0be9f01efe757b42bde35 Nov 02 15:06:37.182924 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.182890 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea31d24-c22c-43da-af1a-e45518e760d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb\" (UID: \"3ea31d24-c22c-43da-af1a-e45518e760d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb" Nov 02 15:06:37.182924 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.182928 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3d924e2-760f-4a18-89bf-3d7e0a73086c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z\" (UID: \"e3d924e2-760f-4a18-89bf-3d7e0a73086c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z" Nov 02 15:06:37.183261 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.183000 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bbqv\" (UniqueName: \"kubernetes.io/projected/e3d924e2-760f-4a18-89bf-3d7e0a73086c-kube-api-access-2bbqv\") pod \"payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z\" (UID: \"e3d924e2-760f-4a18-89bf-3d7e0a73086c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z" Nov 02 15:06:37.183261 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.183053 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk98q\" (UniqueName: \"kubernetes.io/projected/3ea31d24-c22c-43da-af1a-e45518e760d8-kube-api-access-mk98q\") pod \"payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb\" (UID: \"3ea31d24-c22c-43da-af1a-e45518e760d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb" Nov 02 15:06:37.183790 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.183763 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea31d24-c22c-43da-af1a-e45518e760d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb\" (UID: \"3ea31d24-c22c-43da-af1a-e45518e760d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb" Nov 02 15:06:37.183790 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.183778 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3d924e2-760f-4a18-89bf-3d7e0a73086c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z\" (UID: \"e3d924e2-760f-4a18-89bf-3d7e0a73086c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z" Nov 02 15:06:37.207226 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.207202 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk98q\" (UniqueName: \"kubernetes.io/projected/3ea31d24-c22c-43da-af1a-e45518e760d8-kube-api-access-mk98q\") pod \"payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb\" (UID: \"3ea31d24-c22c-43da-af1a-e45518e760d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb" Nov 02 15:06:37.207333 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.207244 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bbqv\" (UniqueName: \"kubernetes.io/projected/e3d924e2-760f-4a18-89bf-3d7e0a73086c-kube-api-access-2bbqv\") pod \"payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z\" (UID: \"e3d924e2-760f-4a18-89bf-3d7e0a73086c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z" Nov 02 15:06:37.257507 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.257466 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" event={"ID":"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a","Type":"ContainerStarted","Data":"f193858818d02cf4d514d172bfba50e6891e5b68e9d0be9f01efe757b42bde35"} Nov 02 15:06:37.265750 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.265725 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb" Nov 02 15:06:37.317594 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.317179 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z" Nov 02 15:06:37.565102 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.565077 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb"] Nov 02 15:06:37.567641 ip-10-0-134-99 kubenswrapper[4866]: W1102 15:06:37.567614 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea31d24_c22c_43da_af1a_e45518e760d8.slice/crio-037dea43e4ea94916fe46b1daf0c03034ebf735582b92a67d91a5cd729d0f31f WatchSource:0}: Error finding container 037dea43e4ea94916fe46b1daf0c03034ebf735582b92a67d91a5cd729d0f31f: Status 404 returned error can't find the container with id 037dea43e4ea94916fe46b1daf0c03034ebf735582b92a67d91a5cd729d0f31f Nov 02 15:06:37.627019 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:37.626988 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z"] Nov 02 15:06:37.630766 ip-10-0-134-99 kubenswrapper[4866]: W1102 15:06:37.630738 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d924e2_760f_4a18_89bf_3d7e0a73086c.slice/crio-ff0b575837cd298c63edf3a8b6952c1581430be12bd04e6358a4ac6ae1f486c4 WatchSource:0}: Error finding container ff0b575837cd298c63edf3a8b6952c1581430be12bd04e6358a4ac6ae1f486c4: Status 404 returned error can't find the container with id ff0b575837cd298c63edf3a8b6952c1581430be12bd04e6358a4ac6ae1f486c4 Nov 02 15:06:38.263657 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:38.263618 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" event={"ID":"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a","Type":"ContainerStarted","Data":"c534adcf58578b9ed68dc423c73e8a4668922cf80160319039c8f9c8807c16b5"} Nov 02 15:06:38.264293 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:38.263688 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" Nov 02 15:06:38.265268 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:38.265243 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb" event={"ID":"3ea31d24-c22c-43da-af1a-e45518e760d8","Type":"ContainerStarted","Data":"e66dd1465f39e86250a0856b5504af3631e49b33b68669c3be3ff3f6a3526e42"} Nov 02 15:06:38.265380 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:38.265293 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb" event={"ID":"3ea31d24-c22c-43da-af1a-e45518e760d8","Type":"ContainerStarted","Data":"037dea43e4ea94916fe46b1daf0c03034ebf735582b92a67d91a5cd729d0f31f"} Nov 02 15:06:38.265380 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:38.265321 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb" Nov 02 15:06:38.266752 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:38.266730 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z" event={"ID":"e3d924e2-760f-4a18-89bf-3d7e0a73086c","Type":"ContainerStarted","Data":"8ffe82de280df52aa54f2f3d6aafc3d3fb33b41e6fd4b08d391a0a58a5fe6b6c"} Nov 02 15:06:38.266844 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:38.266767 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z" event={"ID":"e3d924e2-760f-4a18-89bf-3d7e0a73086c","Type":"ContainerStarted","Data":"ff0b575837cd298c63edf3a8b6952c1581430be12bd04e6358a4ac6ae1f486c4"} Nov 02 15:06:38.266844 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:38.266791 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z" Nov 02 15:06:38.308265 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:38.308210 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" podStartSLOduration=1.745825746 podStartE2EDuration="2.308194783s" totalImagesPullingTime="562.369037ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:38.306781721 +0000 UTC m=+4163.490629060" watchObservedRunningTime="2026-11-02 15:06:38.308194783 +0000 UTC m=+4163.492042122" Nov 02 15:06:38.347034 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:38.346984 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb" podStartSLOduration=2.346968541 podStartE2EDuration="2.346968541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:38.345139399 +0000 UTC m=+4163.528986746" watchObservedRunningTime="2026-11-02 15:06:38.346968541 +0000 UTC m=+4163.530815920" Nov 02 15:06:38.389011 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:38.388941 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z" podStartSLOduration=2.388925613 podStartE2EDuration="2.388925613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:38.387314951 +0000 UTC m=+4163.571162288" watchObservedRunningTime="2026-11-02 15:06:38.388925613 +0000 UTC m=+4163.572772963" Nov 02 15:06:44.278505 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:44.278472 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" Nov 02 15:06:49.275080 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:49.275050 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb" Nov 02 15:06:49.275552 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:49.275152 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z" Nov 02 15:06:55.798856 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:55.798818 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z"] Nov 02 15:06:55.832394 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:55.832357 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb"] Nov 02 15:06:57.795052 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:57.794962 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw"] Nov 02 15:06:57.795477 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:57.795211 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" podUID="ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a" containerName="maas-api" containerID="cri-o://c534adcf58578b9ed68dc423c73e8a4668922cf80160319039c8f9c8807c16b5" gracePeriod=30 Nov 02 15:06:58.049644 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.049565 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" Nov 02 15:06:58.186829 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.186792 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-openshift-service-ca\") pod \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\" (UID: \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\") " Nov 02 15:06:58.187061 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.186842 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-kube-api-access-wcvsg\" (UniqueName: \"kubernetes.io/projected/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-kube-api-access-wcvsg\") pod \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\" (UID: \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\") " Nov 02 15:06:58.187061 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.186984 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-metrics-certs\") pod \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\" (UID: \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\") " Nov 02 15:06:58.187061 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.187036 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-maas-api-tls\") pod \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\" (UID: \"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a\") " Nov 02 15:06:58.187284 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.187256 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-openshift-service-ca" pod "ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a" (UID: "ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:06:58.188991 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.188959 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-kube-api-access-wcvsg" pod "ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a" (UID: "ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a"). InnerVolumeSpecName "kube-api-access-wcvsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:06:58.189105 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.189083 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-metrics-certs" pod "ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a" (UID: "ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:06:58.189147 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.189104 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-maas-api-tls" pod "ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a" (UID: "ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:06:58.288105 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.288064 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:06:58.288105 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.288100 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcvsg\" (UniqueName: \"kubernetes.io/projected/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-kube-api-access-wcvsg\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:06:58.288105 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.288111 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:06:58.288366 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.288120 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:06:58.363876 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.363784 4866 generic.go:356] "Generic (PLEG): container finished" podID="ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a" containerID="c534adcf58578b9ed68dc423c73e8a4668922cf80160319039c8f9c8807c16b5" exitCode=0 Nov 02 15:06:58.363876 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.363839 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" Nov 02 15:06:58.364096 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.363867 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" event={"ID":"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a","Type":"ContainerDied","Data":"c534adcf58578b9ed68dc423c73e8a4668922cf80160319039c8f9c8807c16b5"} Nov 02 15:06:58.364096 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.363923 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw" event={"ID":"ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a","Type":"ContainerDied","Data":"f193858818d02cf4d514d172bfba50e6891e5b68e9d0be9f01efe757b42bde35"} Nov 02 15:06:58.364096 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.363957 4866 scope.go:122] "RemoveContainer" containerID="c534adcf58578b9ed68dc423c73e8a4668922cf80160319039c8f9c8807c16b5" Nov 02 15:06:58.373116 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.373099 4866 scope.go:122] "RemoveContainer" containerID="c534adcf58578b9ed68dc423c73e8a4668922cf80160319039c8f9c8807c16b5" Nov 02 15:06:58.373394 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:06:58.373371 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c534adcf58578b9ed68dc423c73e8a4668922cf80160319039c8f9c8807c16b5\": container with ID starting with c534adcf58578b9ed68dc423c73e8a4668922cf80160319039c8f9c8807c16b5 not found: ID does not exist" containerID="c534adcf58578b9ed68dc423c73e8a4668922cf80160319039c8f9c8807c16b5" Nov 02 15:06:58.373471 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.373400 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c534adcf58578b9ed68dc423c73e8a4668922cf80160319039c8f9c8807c16b5"} err="failed to get container status \"c534adcf58578b9ed68dc423c73e8a4668922cf80160319039c8f9c8807c16b5\": rpc error: code = NotFound desc = could not find container \"c534adcf58578b9ed68dc423c73e8a4668922cf80160319039c8f9c8807c16b5\": container with ID starting with c534adcf58578b9ed68dc423c73e8a4668922cf80160319039c8f9c8807c16b5 not found: ID does not exist" Nov 02 15:06:58.414132 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.414087 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw"] Nov 02 15:06:58.425409 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.425381 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4bc210-6cfb8958bf-dhsbw"] Nov 02 15:06:58.944369 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:06:58.944341 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a" path="/var/lib/kubelet/pods/ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a/volumes" Nov 02 15:07:00.800013 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:00.799942 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z" podUID="e3d924e2-760f-4a18-89bf-3d7e0a73086c" containerName="payload-processing" containerID="cri-o://8ffe82de280df52aa54f2f3d6aafc3d3fb33b41e6fd4b08d391a0a58a5fe6b6c" gracePeriod=25 Nov 02 15:07:00.832686 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:00.832634 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb" podUID="3ea31d24-c22c-43da-af1a-e45518e760d8" containerName="payload-pre-processing" containerID="cri-o://e66dd1465f39e86250a0856b5504af3631e49b33b68669c3be3ff3f6a3526e42" gracePeriod=25 Nov 02 15:07:01.082531 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.082502 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z" Nov 02 15:07:01.112637 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.112614 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb" Nov 02 15:07:01.218316 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.218283 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea31d24-c22c-43da-af1a-e45518e760d8-kube-api-access-mk98q\" (UniqueName: \"kubernetes.io/projected/3ea31d24-c22c-43da-af1a-e45518e760d8-kube-api-access-mk98q\") pod \"3ea31d24-c22c-43da-af1a-e45518e760d8\" (UID: \"3ea31d24-c22c-43da-af1a-e45518e760d8\") " Nov 02 15:07:01.218526 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.218333 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea31d24-c22c-43da-af1a-e45518e760d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea31d24-c22c-43da-af1a-e45518e760d8-plugins-config-volume\") pod \"3ea31d24-c22c-43da-af1a-e45518e760d8\" (UID: \"3ea31d24-c22c-43da-af1a-e45518e760d8\") " Nov 02 15:07:01.218526 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.218393 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3d924e2-760f-4a18-89bf-3d7e0a73086c-kube-api-access-2bbqv\" (UniqueName: \"kubernetes.io/projected/e3d924e2-760f-4a18-89bf-3d7e0a73086c-kube-api-access-2bbqv\") pod \"e3d924e2-760f-4a18-89bf-3d7e0a73086c\" (UID: \"e3d924e2-760f-4a18-89bf-3d7e0a73086c\") " Nov 02 15:07:01.218526 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.218442 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3d924e2-760f-4a18-89bf-3d7e0a73086c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3d924e2-760f-4a18-89bf-3d7e0a73086c-plugins-config-volume\") pod \"e3d924e2-760f-4a18-89bf-3d7e0a73086c\" (UID: \"e3d924e2-760f-4a18-89bf-3d7e0a73086c\") " Nov 02 15:07:01.218853 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.218823 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea31d24-c22c-43da-af1a-e45518e760d8-plugins-config-volume" pod "3ea31d24-c22c-43da-af1a-e45518e760d8" (UID: "3ea31d24-c22c-43da-af1a-e45518e760d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:07:01.218996 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.218862 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d924e2-760f-4a18-89bf-3d7e0a73086c-plugins-config-volume" pod "e3d924e2-760f-4a18-89bf-3d7e0a73086c" (UID: "e3d924e2-760f-4a18-89bf-3d7e0a73086c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:07:01.220551 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.220530 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d924e2-760f-4a18-89bf-3d7e0a73086c-kube-api-access-2bbqv" pod "e3d924e2-760f-4a18-89bf-3d7e0a73086c" (UID: "e3d924e2-760f-4a18-89bf-3d7e0a73086c"). InnerVolumeSpecName "kube-api-access-2bbqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:07:01.220626 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.220556 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea31d24-c22c-43da-af1a-e45518e760d8-kube-api-access-mk98q" pod "3ea31d24-c22c-43da-af1a-e45518e760d8" (UID: "3ea31d24-c22c-43da-af1a-e45518e760d8"). InnerVolumeSpecName "kube-api-access-mk98q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:07:01.319629 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.319539 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3d924e2-760f-4a18-89bf-3d7e0a73086c-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:07:01.319629 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.319572 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk98q\" (UniqueName: \"kubernetes.io/projected/3ea31d24-c22c-43da-af1a-e45518e760d8-kube-api-access-mk98q\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:07:01.319629 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.319586 4866 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea31d24-c22c-43da-af1a-e45518e760d8-plugins-config-volume\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:07:01.319629 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.319598 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bbqv\" (UniqueName: \"kubernetes.io/projected/e3d924e2-760f-4a18-89bf-3d7e0a73086c-kube-api-access-2bbqv\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:07:01.381810 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.381778 4866 generic.go:356] "Generic (PLEG): container finished" podID="3ea31d24-c22c-43da-af1a-e45518e760d8" containerID="e66dd1465f39e86250a0856b5504af3631e49b33b68669c3be3ff3f6a3526e42" exitCode=0 Nov 02 15:07:01.382058 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.381829 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb" Nov 02 15:07:01.382058 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.381873 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb" event={"ID":"3ea31d24-c22c-43da-af1a-e45518e760d8","Type":"ContainerDied","Data":"e66dd1465f39e86250a0856b5504af3631e49b33b68669c3be3ff3f6a3526e42"} Nov 02 15:07:01.382058 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.381934 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb" event={"ID":"3ea31d24-c22c-43da-af1a-e45518e760d8","Type":"ContainerDied","Data":"037dea43e4ea94916fe46b1daf0c03034ebf735582b92a67d91a5cd729d0f31f"} Nov 02 15:07:01.382058 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.381977 4866 scope.go:122] "RemoveContainer" containerID="e66dd1465f39e86250a0856b5504af3631e49b33b68669c3be3ff3f6a3526e42" Nov 02 15:07:01.383401 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.383372 4866 generic.go:356] "Generic (PLEG): container finished" podID="e3d924e2-760f-4a18-89bf-3d7e0a73086c" containerID="8ffe82de280df52aa54f2f3d6aafc3d3fb33b41e6fd4b08d391a0a58a5fe6b6c" exitCode=0 Nov 02 15:07:01.383504 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.383410 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z" event={"ID":"e3d924e2-760f-4a18-89bf-3d7e0a73086c","Type":"ContainerDied","Data":"8ffe82de280df52aa54f2f3d6aafc3d3fb33b41e6fd4b08d391a0a58a5fe6b6c"} Nov 02 15:07:01.383504 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.383419 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z" Nov 02 15:07:01.383504 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.383451 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z" event={"ID":"e3d924e2-760f-4a18-89bf-3d7e0a73086c","Type":"ContainerDied","Data":"ff0b575837cd298c63edf3a8b6952c1581430be12bd04e6358a4ac6ae1f486c4"} Nov 02 15:07:01.397509 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.397475 4866 scope.go:122] "RemoveContainer" containerID="e66dd1465f39e86250a0856b5504af3631e49b33b68669c3be3ff3f6a3526e42" Nov 02 15:07:01.397872 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:07:01.397852 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e66dd1465f39e86250a0856b5504af3631e49b33b68669c3be3ff3f6a3526e42\": container with ID starting with e66dd1465f39e86250a0856b5504af3631e49b33b68669c3be3ff3f6a3526e42 not found: ID does not exist" containerID="e66dd1465f39e86250a0856b5504af3631e49b33b68669c3be3ff3f6a3526e42" Nov 02 15:07:01.397935 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.397880 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e66dd1465f39e86250a0856b5504af3631e49b33b68669c3be3ff3f6a3526e42"} err="failed to get container status \"e66dd1465f39e86250a0856b5504af3631e49b33b68669c3be3ff3f6a3526e42\": rpc error: code = NotFound desc = could not find container \"e66dd1465f39e86250a0856b5504af3631e49b33b68669c3be3ff3f6a3526e42\": container with ID starting with e66dd1465f39e86250a0856b5504af3631e49b33b68669c3be3ff3f6a3526e42 not found: ID does not exist" Nov 02 15:07:01.397935 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.397902 4866 scope.go:122] "RemoveContainer" containerID="8ffe82de280df52aa54f2f3d6aafc3d3fb33b41e6fd4b08d391a0a58a5fe6b6c" Nov 02 15:07:01.406290 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.406265 4866 scope.go:122] "RemoveContainer" containerID="8ffe82de280df52aa54f2f3d6aafc3d3fb33b41e6fd4b08d391a0a58a5fe6b6c" Nov 02 15:07:01.406570 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:07:01.406547 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ffe82de280df52aa54f2f3d6aafc3d3fb33b41e6fd4b08d391a0a58a5fe6b6c\": container with ID starting with 8ffe82de280df52aa54f2f3d6aafc3d3fb33b41e6fd4b08d391a0a58a5fe6b6c not found: ID does not exist" containerID="8ffe82de280df52aa54f2f3d6aafc3d3fb33b41e6fd4b08d391a0a58a5fe6b6c" Nov 02 15:07:01.406623 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.406578 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ffe82de280df52aa54f2f3d6aafc3d3fb33b41e6fd4b08d391a0a58a5fe6b6c"} err="failed to get container status \"8ffe82de280df52aa54f2f3d6aafc3d3fb33b41e6fd4b08d391a0a58a5fe6b6c\": rpc error: code = NotFound desc = could not find container \"8ffe82de280df52aa54f2f3d6aafc3d3fb33b41e6fd4b08d391a0a58a5fe6b6c\": container with ID starting with 8ffe82de280df52aa54f2f3d6aafc3d3fb33b41e6fd4b08d391a0a58a5fe6b6c not found: ID does not exist" Nov 02 15:07:01.432325 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.432288 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb"] Nov 02 15:07:01.436324 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.436296 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4bc210-554c7j7flb"] Nov 02 15:07:01.470069 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.470029 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z"] Nov 02 15:07:01.472123 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:01.472095 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4bc210-f5fbf99cftl55z"] Nov 02 15:07:02.944518 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:02.944486 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea31d24-c22c-43da-af1a-e45518e760d8" path="/var/lib/kubelet/pods/3ea31d24-c22c-43da-af1a-e45518e760d8/volumes" Nov 02 15:07:02.944916 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:02.944852 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3d924e2-760f-4a18-89bf-3d7e0a73086c" path="/var/lib/kubelet/pods/e3d924e2-760f-4a18-89bf-3d7e0a73086c/volumes" Nov 02 15:07:06.008772 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:07:06.008732 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:07:15.730115 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:15.730086 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 15:07:15.751500 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:15.751475 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 15:07:33.579164 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.579125 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn"] Nov 02 15:07:33.579751 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.579733 4866 cpu_manager.go:414] "containerMap: removing container" podUID="ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a" containerName="maas-api" Nov 02 15:07:33.579751 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.579750 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a" containerName="maas-api" Nov 02 15:07:33.579877 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.579762 4866 cpu_manager.go:414] "containerMap: removing container" podUID="3ea31d24-c22c-43da-af1a-e45518e760d8" containerName="payload-pre-processing" Nov 02 15:07:33.579877 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.579768 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea31d24-c22c-43da-af1a-e45518e760d8" containerName="payload-pre-processing" Nov 02 15:07:33.579877 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.579786 4866 cpu_manager.go:414] "containerMap: removing container" podUID="e3d924e2-760f-4a18-89bf-3d7e0a73086c" containerName="payload-processing" Nov 02 15:07:33.579877 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.579791 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3d924e2-760f-4a18-89bf-3d7e0a73086c" containerName="payload-processing" Nov 02 15:07:33.579877 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.579857 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3d924e2-760f-4a18-89bf-3d7e0a73086c" containerName="payload-processing" Nov 02 15:07:33.579877 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.579869 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce1f25eb-1a57-45b8-9afd-169ad9d6ea0a" containerName="maas-api" Nov 02 15:07:33.579877 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.579875 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea31d24-c22c-43da-af1a-e45518e760d8" containerName="payload-pre-processing" Nov 02 15:07:33.582971 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.582937 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn" Nov 02 15:07:33.605504 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.605449 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn"] Nov 02 15:07:33.762238 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.762205 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03db21be-fe6e-46ce-8ab1-09ab2569ac1d-maas-api-tls\") pod \"maas-api-5956f6dcff-d2rtn\" (UID: \"03db21be-fe6e-46ce-8ab1-09ab2569ac1d\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn" Nov 02 15:07:33.762448 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.762268 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03db21be-fe6e-46ce-8ab1-09ab2569ac1d-metrics-certs\") pod \"maas-api-5956f6dcff-d2rtn\" (UID: \"03db21be-fe6e-46ce-8ab1-09ab2569ac1d\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn" Nov 02 15:07:33.762448 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.762393 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03db21be-fe6e-46ce-8ab1-09ab2569ac1d-openshift-service-ca\") pod \"maas-api-5956f6dcff-d2rtn\" (UID: \"03db21be-fe6e-46ce-8ab1-09ab2569ac1d\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn" Nov 02 15:07:33.762573 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.762461 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skpnm\" (UniqueName: \"kubernetes.io/projected/03db21be-fe6e-46ce-8ab1-09ab2569ac1d-kube-api-access-skpnm\") pod \"maas-api-5956f6dcff-d2rtn\" (UID: \"03db21be-fe6e-46ce-8ab1-09ab2569ac1d\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn" Nov 02 15:07:33.863596 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.863504 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03db21be-fe6e-46ce-8ab1-09ab2569ac1d-metrics-certs\") pod \"maas-api-5956f6dcff-d2rtn\" (UID: \"03db21be-fe6e-46ce-8ab1-09ab2569ac1d\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn" Nov 02 15:07:33.863596 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.863560 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03db21be-fe6e-46ce-8ab1-09ab2569ac1d-openshift-service-ca\") pod \"maas-api-5956f6dcff-d2rtn\" (UID: \"03db21be-fe6e-46ce-8ab1-09ab2569ac1d\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn" Nov 02 15:07:33.863846 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.863601 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skpnm\" (UniqueName: \"kubernetes.io/projected/03db21be-fe6e-46ce-8ab1-09ab2569ac1d-kube-api-access-skpnm\") pod \"maas-api-5956f6dcff-d2rtn\" (UID: \"03db21be-fe6e-46ce-8ab1-09ab2569ac1d\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn" Nov 02 15:07:33.863846 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.863633 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03db21be-fe6e-46ce-8ab1-09ab2569ac1d-maas-api-tls\") pod \"maas-api-5956f6dcff-d2rtn\" (UID: \"03db21be-fe6e-46ce-8ab1-09ab2569ac1d\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn" Nov 02 15:07:33.864392 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.864370 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03db21be-fe6e-46ce-8ab1-09ab2569ac1d-openshift-service-ca\") pod \"maas-api-5956f6dcff-d2rtn\" (UID: \"03db21be-fe6e-46ce-8ab1-09ab2569ac1d\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn" Nov 02 15:07:33.866206 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.866177 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03db21be-fe6e-46ce-8ab1-09ab2569ac1d-metrics-certs\") pod \"maas-api-5956f6dcff-d2rtn\" (UID: \"03db21be-fe6e-46ce-8ab1-09ab2569ac1d\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn" Nov 02 15:07:33.866312 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.866184 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03db21be-fe6e-46ce-8ab1-09ab2569ac1d-maas-api-tls\") pod \"maas-api-5956f6dcff-d2rtn\" (UID: \"03db21be-fe6e-46ce-8ab1-09ab2569ac1d\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn" Nov 02 15:07:33.886475 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.886444 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skpnm\" (UniqueName: \"kubernetes.io/projected/03db21be-fe6e-46ce-8ab1-09ab2569ac1d-kube-api-access-skpnm\") pod \"maas-api-5956f6dcff-d2rtn\" (UID: \"03db21be-fe6e-46ce-8ab1-09ab2569ac1d\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn" Nov 02 15:07:33.896426 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:33.896403 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn" Nov 02 15:07:34.193075 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:34.192933 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn"] Nov 02 15:07:34.552474 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:34.552391 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn" event={"ID":"03db21be-fe6e-46ce-8ab1-09ab2569ac1d","Type":"ContainerStarted","Data":"fae25f8f4dd17ad551015c09bfb8ecacc7fa2651153d7f08b5c73b941c94f008"} Nov 02 15:07:35.559749 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:35.559709 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn" event={"ID":"03db21be-fe6e-46ce-8ab1-09ab2569ac1d","Type":"ContainerStarted","Data":"44dbb587736f704a1ae25f1dbddf87b629109051f92fe29146d82b9374ca91d8"} Nov 02 15:07:35.560562 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:35.559801 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn" Nov 02 15:07:35.602351 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:35.602288 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn" podStartSLOduration=2.224594397 podStartE2EDuration="2.602275104s" totalImagesPullingTime="377.680707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:35.600539788 +0000 UTC m=+4220.784387127" watchObservedRunningTime="2026-11-02 15:07:35.602275104 +0000 UTC m=+4220.786122440" Nov 02 15:07:36.010078 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:07:36.010042 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:07:41.570340 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:41.570308 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5956f6dcff-d2rtn" Nov 02 15:07:41.649790 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:41.649758 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr"] Nov 02 15:07:41.650063 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:41.650034 4866 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" podUID="a1f38a69-4298-4c7c-a287-190d35223473" containerName="maas-api" containerID="cri-o://160b6b2f96efd31b52d4726ed54973061dbedd6ba818aae0d315b095136c0a42" gracePeriod=30 Nov 02 15:07:42.909430 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:42.909405 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" Nov 02 15:07:42.948108 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:42.948082 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1f38a69-4298-4c7c-a287-190d35223473-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1f38a69-4298-4c7c-a287-190d35223473-maas-api-tls\") pod \"a1f38a69-4298-4c7c-a287-190d35223473\" (UID: \"a1f38a69-4298-4c7c-a287-190d35223473\") " Nov 02 15:07:42.948260 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:42.948135 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1f38a69-4298-4c7c-a287-190d35223473-kube-api-access-hdmh6\" (UniqueName: \"kubernetes.io/projected/a1f38a69-4298-4c7c-a287-190d35223473-kube-api-access-hdmh6\") pod \"a1f38a69-4298-4c7c-a287-190d35223473\" (UID: \"a1f38a69-4298-4c7c-a287-190d35223473\") " Nov 02 15:07:42.948260 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:42.948200 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1f38a69-4298-4c7c-a287-190d35223473-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f38a69-4298-4c7c-a287-190d35223473-metrics-certs\") pod \"a1f38a69-4298-4c7c-a287-190d35223473\" (UID: \"a1f38a69-4298-4c7c-a287-190d35223473\") " Nov 02 15:07:42.948260 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:42.948231 4866 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1f38a69-4298-4c7c-a287-190d35223473-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f38a69-4298-4c7c-a287-190d35223473-openshift-service-ca\") pod \"a1f38a69-4298-4c7c-a287-190d35223473\" (UID: \"a1f38a69-4298-4c7c-a287-190d35223473\") " Nov 02 15:07:42.948677 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:42.948643 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1f38a69-4298-4c7c-a287-190d35223473-openshift-service-ca" pod "a1f38a69-4298-4c7c-a287-190d35223473" (UID: "a1f38a69-4298-4c7c-a287-190d35223473"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:07:42.950286 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:42.950254 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1f38a69-4298-4c7c-a287-190d35223473-maas-api-tls" pod "a1f38a69-4298-4c7c-a287-190d35223473" (UID: "a1f38a69-4298-4c7c-a287-190d35223473"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:07:42.950394 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:42.950369 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f38a69-4298-4c7c-a287-190d35223473-kube-api-access-hdmh6" pod "a1f38a69-4298-4c7c-a287-190d35223473" (UID: "a1f38a69-4298-4c7c-a287-190d35223473"). InnerVolumeSpecName "kube-api-access-hdmh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:07:42.950481 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:42.950461 4866 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1f38a69-4298-4c7c-a287-190d35223473-metrics-certs" pod "a1f38a69-4298-4c7c-a287-190d35223473" (UID: "a1f38a69-4298-4c7c-a287-190d35223473"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:07:43.049271 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:43.049182 4866 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f38a69-4298-4c7c-a287-190d35223473-metrics-certs\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:07:43.049271 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:43.049215 4866 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f38a69-4298-4c7c-a287-190d35223473-openshift-service-ca\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:07:43.049271 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:43.049227 4866 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1f38a69-4298-4c7c-a287-190d35223473-maas-api-tls\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:07:43.049271 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:43.049237 4866 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdmh6\" (UniqueName: \"kubernetes.io/projected/a1f38a69-4298-4c7c-a287-190d35223473-kube-api-access-hdmh6\") on node \"ip-10-0-134-99.ec2.internal\" DevicePath \"\"" Nov 02 15:07:43.598811 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:43.598774 4866 generic.go:356] "Generic (PLEG): container finished" podID="a1f38a69-4298-4c7c-a287-190d35223473" containerID="160b6b2f96efd31b52d4726ed54973061dbedd6ba818aae0d315b095136c0a42" exitCode=0 Nov 02 15:07:43.599043 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:43.598821 4866 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" Nov 02 15:07:43.599043 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:43.598855 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" event={"ID":"a1f38a69-4298-4c7c-a287-190d35223473","Type":"ContainerDied","Data":"160b6b2f96efd31b52d4726ed54973061dbedd6ba818aae0d315b095136c0a42"} Nov 02 15:07:43.599043 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:43.598903 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr" event={"ID":"a1f38a69-4298-4c7c-a287-190d35223473","Type":"ContainerDied","Data":"24e084e5e65ede40f6d8d799f32255c7079c821b8c6e0aa7de9a8ef861bc1697"} Nov 02 15:07:43.599043 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:43.598924 4866 scope.go:122] "RemoveContainer" containerID="160b6b2f96efd31b52d4726ed54973061dbedd6ba818aae0d315b095136c0a42" Nov 02 15:07:43.608385 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:43.608362 4866 scope.go:122] "RemoveContainer" containerID="160b6b2f96efd31b52d4726ed54973061dbedd6ba818aae0d315b095136c0a42" Nov 02 15:07:43.608658 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:07:43.608639 4866 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"160b6b2f96efd31b52d4726ed54973061dbedd6ba818aae0d315b095136c0a42\": container with ID starting with 160b6b2f96efd31b52d4726ed54973061dbedd6ba818aae0d315b095136c0a42 not found: ID does not exist" containerID="160b6b2f96efd31b52d4726ed54973061dbedd6ba818aae0d315b095136c0a42" Nov 02 15:07:43.608709 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:43.608667 4866 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"160b6b2f96efd31b52d4726ed54973061dbedd6ba818aae0d315b095136c0a42"} err="failed to get container status \"160b6b2f96efd31b52d4726ed54973061dbedd6ba818aae0d315b095136c0a42\": rpc error: code = NotFound desc = could not find container \"160b6b2f96efd31b52d4726ed54973061dbedd6ba818aae0d315b095136c0a42\": container with ID starting with 160b6b2f96efd31b52d4726ed54973061dbedd6ba818aae0d315b095136c0a42 not found: ID does not exist" Nov 02 15:07:43.651508 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:43.651478 4866 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr"] Nov 02 15:07:43.663342 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:43.663316 4866 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-55b5486c4c-dwrbr"] Nov 02 15:07:44.944509 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:07:44.944473 4866 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1f38a69-4298-4c7c-a287-190d35223473" path="/var/lib/kubelet/pods/a1f38a69-4298-4c7c-a287-190d35223473/volumes" Nov 02 15:08:06.009256 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:08:06.009216 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:08:36.009309 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:08:36.009230 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:09:06.008824 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:09:06.008787 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:09:36.009406 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:09:36.009355 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:10:06.009015 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:10:06.008922 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:10:36.010802 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:10:36.010686 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:10:37.498295 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:10:37.498262 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64c547bcd4-s7w4p_a9a48536-da8d-42b1-ac12-d5addcc2cadc/authorino/0.log" Nov 02 15:10:40.657174 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:10:40.657125 4866 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-99.ec2.internal" podUID="fcd82cc88301759f1e08e2b014a50651" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:10:59.646908 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:10:59.646874 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6cc7b895d5-5ldn8_4865da86-417f-4703-bcdc-c68b5de3ad59/manager/0.log" Nov 02 15:11:01.470213 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:01.470182 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-c2cbh_36201127-d7ae-4a96-879f-d28da880063c/proxy/0.log" Nov 02 15:11:02.638620 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:02.638592 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5956f6dcff-d2rtn_03db21be-fe6e-46ce-8ab1-09ab2569ac1d/maas-api/0.log" Nov 02 15:11:06.009551 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:11:06.009512 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:11:26.502038 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:26.502005 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64c547bcd4-s7w4p_a9a48536-da8d-42b1-ac12-d5addcc2cadc/authorino/0.log" Nov 02 15:11:29.363501 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:29.363426 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8f9c48f5f-jlflv_67f443f5-5750-4236-bcf7-b3239b253c9e/kube-auth-proxy/0.log" Nov 02 15:11:29.773602 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:29.773569 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-259d8_cd410bed-b4ef-467a-95f9-9b520abfd44d/istio-proxy/0.log" Nov 02 15:11:30.609142 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:30.609108 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-8q5hv_1fbf701c-a1aa-4031-8112-4e69e00e103b/istio-proxy/0.log" Nov 02 15:11:30.827205 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:30.827173 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-9pqlk_e3459850-2b2b-4dc0-b1ab-5ffbc87263a6/istio-proxy/0.log" Nov 02 15:11:32.141645 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:32.141617 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-bbc4b5f5d-l54ph_3ce257ea-3be4-412a-92b5-cc93696643c0/payload-pre-processing/0.log" Nov 02 15:11:32.342972 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:32.342917 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-8f7f4697c-89t4w_dc87147a-63b6-4b08-b45a-e90793367a36/payload-processing/0.log" Nov 02 15:11:32.536078 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:32.536048 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7899fcdf65-hl87t_87b5876f-0012-4bdf-b037-25070684476c/router/0.log" Nov 02 15:11:33.594483 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:33.594448 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-hlvsg_c8983367-2f54-4d0c-879a-bc768a47a1ee/main/0.log" Nov 02 15:11:33.804745 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:33.804707 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-kwrdx_d2f9dd22-8f5b-477f-9ca6-3ea2a6e9b102/main/0.log" Nov 02 15:11:34.192365 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:34.192338 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-nnfxn_4ba45fd0-b74d-4b8c-be7a-4d521845c7d4/main/0.log" Nov 02 15:11:34.594919 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:34.594842 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-92h45_c9ef4054-ab42-4598-a5e1-ac424fab8a8e/main/0.log" Nov 02 15:11:34.989573 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:34.989543 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-g65gv_6d89a28a-7f87-4924-93c3-48ae7d11e56e/main/0.log" Nov 02 15:11:36.010353 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:11:36.010319 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:11:45.995503 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:45.995456 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l7lm6/must-gather-hvhs4"] Nov 02 15:11:45.996117 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:45.996033 4866 cpu_manager.go:414] "containerMap: removing container" podUID="a1f38a69-4298-4c7c-a287-190d35223473" containerName="maas-api" Nov 02 15:11:45.996117 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:45.996053 4866 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1f38a69-4298-4c7c-a287-190d35223473" containerName="maas-api" Nov 02 15:11:45.996245 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:45.996230 4866 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1f38a69-4298-4c7c-a287-190d35223473" containerName="maas-api" Nov 02 15:11:45.999391 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:45.999374 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7lm6/must-gather-hvhs4" Nov 02 15:11:46.013785 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.013757 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l7lm6\"/\"openshift-service-ca.crt\"" Nov 02 15:11:46.013924 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.013771 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l7lm6\"/\"kube-root-ca.crt\"" Nov 02 15:11:46.021647 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.021621 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b6bbd407-e718-4393-b55f-f79590a7cb5e-must-gather-output\") pod \"must-gather-hvhs4\" (UID: \"b6bbd407-e718-4393-b55f-f79590a7cb5e\") " pod="openshift-must-gather-l7lm6/must-gather-hvhs4" Nov 02 15:11:46.021782 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.021764 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgt6b\" (UniqueName: \"kubernetes.io/projected/b6bbd407-e718-4393-b55f-f79590a7cb5e-kube-api-access-fgt6b\") pod \"must-gather-hvhs4\" (UID: \"b6bbd407-e718-4393-b55f-f79590a7cb5e\") " pod="openshift-must-gather-l7lm6/must-gather-hvhs4" Nov 02 15:11:46.025367 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.025340 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l7lm6\"/\"default-dockercfg-stbfl\"" Nov 02 15:11:46.030862 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.030830 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7lm6/must-gather-hvhs4"] Nov 02 15:11:46.085247 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.085215 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fcxf8/must-gather-8mgtq"] Nov 02 15:11:46.088813 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.088795 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fcxf8/must-gather-8mgtq" Nov 02 15:11:46.111846 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.111815 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fcxf8/must-gather-8mgtq"] Nov 02 15:11:46.114042 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.114012 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fcxf8\"/\"kube-root-ca.crt\"" Nov 02 15:11:46.114194 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.114099 4866 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fcxf8\"/\"default-dockercfg-pmlxg\"" Nov 02 15:11:46.114194 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.114135 4866 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fcxf8\"/\"openshift-service-ca.crt\"" Nov 02 15:11:46.122841 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.122805 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgt6b\" (UniqueName: \"kubernetes.io/projected/b6bbd407-e718-4393-b55f-f79590a7cb5e-kube-api-access-fgt6b\") pod \"must-gather-hvhs4\" (UID: \"b6bbd407-e718-4393-b55f-f79590a7cb5e\") " pod="openshift-must-gather-l7lm6/must-gather-hvhs4" Nov 02 15:11:46.123042 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.122903 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b6bbd407-e718-4393-b55f-f79590a7cb5e-must-gather-output\") pod \"must-gather-hvhs4\" (UID: \"b6bbd407-e718-4393-b55f-f79590a7cb5e\") " pod="openshift-must-gather-l7lm6/must-gather-hvhs4" Nov 02 15:11:46.123042 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.122937 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8d516969-9630-47c9-affe-c82f38c17abb-must-gather-output\") pod \"must-gather-8mgtq\" (UID: \"8d516969-9630-47c9-affe-c82f38c17abb\") " pod="openshift-must-gather-fcxf8/must-gather-8mgtq" Nov 02 15:11:46.123166 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.123040 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8krn6\" (UniqueName: \"kubernetes.io/projected/8d516969-9630-47c9-affe-c82f38c17abb-kube-api-access-8krn6\") pod \"must-gather-8mgtq\" (UID: \"8d516969-9630-47c9-affe-c82f38c17abb\") " pod="openshift-must-gather-fcxf8/must-gather-8mgtq" Nov 02 15:11:46.123266 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.123247 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b6bbd407-e718-4393-b55f-f79590a7cb5e-must-gather-output\") pod \"must-gather-hvhs4\" (UID: \"b6bbd407-e718-4393-b55f-f79590a7cb5e\") " pod="openshift-must-gather-l7lm6/must-gather-hvhs4" Nov 02 15:11:46.146626 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.146590 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgt6b\" (UniqueName: \"kubernetes.io/projected/b6bbd407-e718-4393-b55f-f79590a7cb5e-kube-api-access-fgt6b\") pod \"must-gather-hvhs4\" (UID: \"b6bbd407-e718-4393-b55f-f79590a7cb5e\") " pod="openshift-must-gather-l7lm6/must-gather-hvhs4" Nov 02 15:11:46.224385 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.224344 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8d516969-9630-47c9-affe-c82f38c17abb-must-gather-output\") pod \"must-gather-8mgtq\" (UID: \"8d516969-9630-47c9-affe-c82f38c17abb\") " pod="openshift-must-gather-fcxf8/must-gather-8mgtq" Nov 02 15:11:46.224572 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.224431 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8krn6\" (UniqueName: \"kubernetes.io/projected/8d516969-9630-47c9-affe-c82f38c17abb-kube-api-access-8krn6\") pod \"must-gather-8mgtq\" (UID: \"8d516969-9630-47c9-affe-c82f38c17abb\") " pod="openshift-must-gather-fcxf8/must-gather-8mgtq" Nov 02 15:11:46.224747 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.224725 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8d516969-9630-47c9-affe-c82f38c17abb-must-gather-output\") pod \"must-gather-8mgtq\" (UID: \"8d516969-9630-47c9-affe-c82f38c17abb\") " pod="openshift-must-gather-fcxf8/must-gather-8mgtq" Nov 02 15:11:46.247369 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.247307 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8krn6\" (UniqueName: \"kubernetes.io/projected/8d516969-9630-47c9-affe-c82f38c17abb-kube-api-access-8krn6\") pod \"must-gather-8mgtq\" (UID: \"8d516969-9630-47c9-affe-c82f38c17abb\") " pod="openshift-must-gather-fcxf8/must-gather-8mgtq" Nov 02 15:11:46.314314 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.314275 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7lm6/must-gather-hvhs4" Nov 02 15:11:46.402317 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.402284 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fcxf8/must-gather-8mgtq" Nov 02 15:11:46.609467 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.609429 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7lm6/must-gather-hvhs4"] Nov 02 15:11:46.611738 ip-10-0-134-99 kubenswrapper[4866]: W1102 15:11:46.611710 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6bbd407_e718_4393_b55f_f79590a7cb5e.slice/crio-f171bee55d78ae2d51dd96107a82b91998254db91cec0217388f4bbe96a92959 WatchSource:0}: Error finding container f171bee55d78ae2d51dd96107a82b91998254db91cec0217388f4bbe96a92959: Status 404 returned error can't find the container with id f171bee55d78ae2d51dd96107a82b91998254db91cec0217388f4bbe96a92959 Nov 02 15:11:46.613610 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.613586 4866 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:11:46.698827 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.698795 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fcxf8/must-gather-8mgtq"] Nov 02 15:11:46.701501 ip-10-0-134-99 kubenswrapper[4866]: W1102 15:11:46.701472 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d516969_9630_47c9_affe_c82f38c17abb.slice/crio-021e3ad27363285de18d2f6cf9bd3efee33cf809f44be6617dc287d3f7ca6775 WatchSource:0}: Error finding container 021e3ad27363285de18d2f6cf9bd3efee33cf809f44be6617dc287d3f7ca6775: Status 404 returned error can't find the container with id 021e3ad27363285de18d2f6cf9bd3efee33cf809f44be6617dc287d3f7ca6775 Nov 02 15:11:46.886015 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.885908 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcxf8/must-gather-8mgtq" event={"ID":"8d516969-9630-47c9-affe-c82f38c17abb","Type":"ContainerStarted","Data":"021e3ad27363285de18d2f6cf9bd3efee33cf809f44be6617dc287d3f7ca6775"} Nov 02 15:11:46.887045 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:46.887011 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7lm6/must-gather-hvhs4" event={"ID":"b6bbd407-e718-4393-b55f-f79590a7cb5e","Type":"ContainerStarted","Data":"f171bee55d78ae2d51dd96107a82b91998254db91cec0217388f4bbe96a92959"} Nov 02 15:11:47.897118 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:47.896374 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcxf8/must-gather-8mgtq" event={"ID":"8d516969-9630-47c9-affe-c82f38c17abb","Type":"ContainerStarted","Data":"2ff2969f760bdccf19c7ba45ee17ccf64ee95202c66725ce9e5106d922cf97bc"} Nov 02 15:11:47.897118 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:47.896435 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcxf8/must-gather-8mgtq" event={"ID":"8d516969-9630-47c9-affe-c82f38c17abb","Type":"ContainerStarted","Data":"e006e29ce4dbc92ca00676d58d570aae36e7456bb926166a56845292beb3ac8b"} Nov 02 15:11:47.946267 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:47.946207 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fcxf8/must-gather-8mgtq" podStartSLOduration=1.27279689 podStartE2EDuration="1.946187983s" totalImagesPullingTime="673.391093ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:47.944411613 +0000 UTC m=+4473.128258957" watchObservedRunningTime="2026-11-02 15:11:47.946187983 +0000 UTC m=+4473.130035319" Nov 02 15:11:51.342709 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:51.342670 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kv25k_d1e29491-8fb3-4918-b930-796653f412ec/global-pull-secret-syncer/0.log" Nov 02 15:11:51.738021 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:51.737994 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-clfrl_7157562b-09b1-44a1-95f8-f29c7e62e475/connection-checker/0.log" Nov 02 15:11:51.872545 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:51.872514 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-frx4v_a7095b0b-813f-4b60-989f-8a01e2743bbe/connection-checker/0.log" Nov 02 15:11:52.037348 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:52.028311 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-mbpqq_d445fa88-f52d-460f-b89a-61de9cc8d541/connection-checker/0.log" Nov 02 15:11:52.530968 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:52.530443 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5vmgp_aa6a722d-ff18-4837-8169-f1a642a268bb/konnectivity-agent/0.log" Nov 02 15:11:52.956965 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:52.953159 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-99.ec2.internal_fcd82cc88301759f1e08e2b014a50651/haproxy/0.log" Nov 02 15:11:57.039975 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:57.034428 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7lm6/must-gather-hvhs4" event={"ID":"b6bbd407-e718-4393-b55f-f79590a7cb5e","Type":"ContainerStarted","Data":"8c00cd4938e046cc36b39d68fb6dcb4f0b36ce2b577c47246e0c162159c96db3"} Nov 02 15:11:57.039975 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:57.034488 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7lm6/must-gather-hvhs4" event={"ID":"b6bbd407-e718-4393-b55f-f79590a7cb5e","Type":"ContainerStarted","Data":"ed3c9c7cc70e52d2b9ce278d1aa034a67c3a7be828f2b94024958f933bdc1a7a"} Nov 02 15:11:57.084973 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:57.084118 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-l7lm6/must-gather-hvhs4" podStartSLOduration=3.020719177 podStartE2EDuration="12.084096308s" totalImagesPullingTime="9.063377131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:57.080887126 +0000 UTC m=+4482.264734464" watchObservedRunningTime="2026-11-02 15:11:57.084096308 +0000 UTC m=+4482.267943646" Nov 02 15:11:59.006187 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:11:59.006114 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64c547bcd4-s7w4p_a9a48536-da8d-42b1-ac12-d5addcc2cadc/authorino/0.log" Nov 02 15:12:01.705492 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:01.704301 4866 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9"] Nov 02 15:12:01.708226 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:01.708194 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" Nov 02 15:12:01.731103 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:01.731072 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9"] Nov 02 15:12:01.847482 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:01.847431 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9a5fd36-4b60-4a11-96ff-109b6e271a99-lib-modules\") pod \"perf-node-gather-daemonset-54fj9\" (UID: \"e9a5fd36-4b60-4a11-96ff-109b6e271a99\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" Nov 02 15:12:01.847482 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:01.847481 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e9a5fd36-4b60-4a11-96ff-109b6e271a99-proc\") pod \"perf-node-gather-daemonset-54fj9\" (UID: \"e9a5fd36-4b60-4a11-96ff-109b6e271a99\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" Nov 02 15:12:01.847745 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:01.847696 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6ct4\" (UniqueName: \"kubernetes.io/projected/e9a5fd36-4b60-4a11-96ff-109b6e271a99-kube-api-access-x6ct4\") pod \"perf-node-gather-daemonset-54fj9\" (UID: \"e9a5fd36-4b60-4a11-96ff-109b6e271a99\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" Nov 02 15:12:01.847806 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:01.847756 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e9a5fd36-4b60-4a11-96ff-109b6e271a99-podres\") pod \"perf-node-gather-daemonset-54fj9\" (UID: \"e9a5fd36-4b60-4a11-96ff-109b6e271a99\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" Nov 02 15:12:01.847879 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:01.847857 4866 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9a5fd36-4b60-4a11-96ff-109b6e271a99-sys\") pod \"perf-node-gather-daemonset-54fj9\" (UID: \"e9a5fd36-4b60-4a11-96ff-109b6e271a99\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" Nov 02 15:12:01.948937 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:01.948899 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ct4\" (UniqueName: \"kubernetes.io/projected/e9a5fd36-4b60-4a11-96ff-109b6e271a99-kube-api-access-x6ct4\") pod \"perf-node-gather-daemonset-54fj9\" (UID: \"e9a5fd36-4b60-4a11-96ff-109b6e271a99\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" Nov 02 15:12:01.949132 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:01.948989 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e9a5fd36-4b60-4a11-96ff-109b6e271a99-podres\") pod \"perf-node-gather-daemonset-54fj9\" (UID: \"e9a5fd36-4b60-4a11-96ff-109b6e271a99\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" Nov 02 15:12:01.949132 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:01.949021 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9a5fd36-4b60-4a11-96ff-109b6e271a99-sys\") pod \"perf-node-gather-daemonset-54fj9\" (UID: \"e9a5fd36-4b60-4a11-96ff-109b6e271a99\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" Nov 02 15:12:01.949132 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:01.949050 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9a5fd36-4b60-4a11-96ff-109b6e271a99-lib-modules\") pod \"perf-node-gather-daemonset-54fj9\" (UID: \"e9a5fd36-4b60-4a11-96ff-109b6e271a99\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" Nov 02 15:12:01.949132 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:01.949131 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9a5fd36-4b60-4a11-96ff-109b6e271a99-lib-modules\") pod \"perf-node-gather-daemonset-54fj9\" (UID: \"e9a5fd36-4b60-4a11-96ff-109b6e271a99\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" Nov 02 15:12:01.949332 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:01.949128 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e9a5fd36-4b60-4a11-96ff-109b6e271a99-podres\") pod \"perf-node-gather-daemonset-54fj9\" (UID: \"e9a5fd36-4b60-4a11-96ff-109b6e271a99\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" Nov 02 15:12:01.949332 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:01.949128 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9a5fd36-4b60-4a11-96ff-109b6e271a99-sys\") pod \"perf-node-gather-daemonset-54fj9\" (UID: \"e9a5fd36-4b60-4a11-96ff-109b6e271a99\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" Nov 02 15:12:01.949332 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:01.949169 4866 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e9a5fd36-4b60-4a11-96ff-109b6e271a99-proc\") pod \"perf-node-gather-daemonset-54fj9\" (UID: \"e9a5fd36-4b60-4a11-96ff-109b6e271a99\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" Nov 02 15:12:01.949332 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:01.949273 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e9a5fd36-4b60-4a11-96ff-109b6e271a99-proc\") pod \"perf-node-gather-daemonset-54fj9\" (UID: \"e9a5fd36-4b60-4a11-96ff-109b6e271a99\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" Nov 02 15:12:01.972734 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:01.972662 4866 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6ct4\" (UniqueName: \"kubernetes.io/projected/e9a5fd36-4b60-4a11-96ff-109b6e271a99-kube-api-access-x6ct4\") pod \"perf-node-gather-daemonset-54fj9\" (UID: \"e9a5fd36-4b60-4a11-96ff-109b6e271a99\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" Nov 02 15:12:02.026707 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:02.026670 4866 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" Nov 02 15:12:02.341514 ip-10-0-134-99 kubenswrapper[4866]: W1102 15:12:02.341482 4866 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode9a5fd36_4b60_4a11_96ff_109b6e271a99.slice/crio-2125c6fb2f32a6d2e8a38582032a7f9d6be7d64d50945c7e90566e99aad55036 WatchSource:0}: Error finding container 2125c6fb2f32a6d2e8a38582032a7f9d6be7d64d50945c7e90566e99aad55036: Status 404 returned error can't find the container with id 2125c6fb2f32a6d2e8a38582032a7f9d6be7d64d50945c7e90566e99aad55036 Nov 02 15:12:02.343195 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:02.343168 4866 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9"] Nov 02 15:12:02.857332 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:02.857295 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-4tgbq_b1259f5f-470e-4ca5-8760-37bff1735835/cluster-monitoring-operator/0.log" Nov 02 15:12:03.083309 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:03.083270 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" event={"ID":"e9a5fd36-4b60-4a11-96ff-109b6e271a99","Type":"ContainerStarted","Data":"afe8a929cdc529b0ca209b84d3c896ceadc6ab79488bc5e1fd6063cb48b56de6"} Nov 02 15:12:03.083564 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:03.083344 4866 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" event={"ID":"e9a5fd36-4b60-4a11-96ff-109b6e271a99","Type":"ContainerStarted","Data":"2125c6fb2f32a6d2e8a38582032a7f9d6be7d64d50945c7e90566e99aad55036"} Nov 02 15:12:03.083564 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:03.083377 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" Nov 02 15:12:03.126680 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:03.126549 4866 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" podStartSLOduration=2.126527466 podStartE2EDuration="2.126527466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 15:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:03.123257207 +0000 UTC m=+4488.307104544" watchObservedRunningTime="2026-11-02 15:12:03.126527466 +0000 UTC m=+4488.310374807" Nov 02 15:12:04.024096 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:04.024067 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hd8x9_a80e1d69-fa23-46db-88f9-18fd14912fa8/node-exporter/0.log" Nov 02 15:12:04.151748 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:04.151722 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hd8x9_a80e1d69-fa23-46db-88f9-18fd14912fa8/kube-rbac-proxy/0.log" Nov 02 15:12:04.279789 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:04.279711 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hd8x9_a80e1d69-fa23-46db-88f9-18fd14912fa8/init-textfile/0.log" Nov 02 15:12:06.009602 ip-10-0-134-99 kubenswrapper[4866]: E1102 15:12:06.009569 4866 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6kpv" containerName="extractor" Nov 02 15:12:08.364319 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:08.364284 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6cc7b895d5-5ldn8_4865da86-417f-4703-bcdc-c68b5de3ad59/manager/0.log" Nov 02 15:12:08.952806 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:08.952772 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-hvxsl_cbd289ce-fac0-4e37-a9e8-c8b1cc8fa4c1/networking-console-plugin/0.log" Nov 02 15:12:09.100452 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:09.100421 4866 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-54fj9" Nov 02 15:12:10.142453 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:10.142420 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 15:12:10.148580 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:10.148542 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/3.log" Nov 02 15:12:10.156358 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:10.156318 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-c2cbh_36201127-d7ae-4a96-879f-d28da880063c/proxy/0.log" Nov 02 15:12:11.820691 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:11.820658 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6cc7b895d5-5ldn8_4865da86-417f-4703-bcdc-c68b5de3ad59/manager/0.log" Nov 02 15:12:12.605069 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:12.605007 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-xrfvm_cf94ad7e-96a0-4d35-9cb3-ad83f8234281/volume-data-source-validator/0.log" Nov 02 15:12:13.691235 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:13.691201 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-c2cbh_36201127-d7ae-4a96-879f-d28da880063c/proxy/0.log" Nov 02 15:12:15.105195 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:15.105159 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6cc7b895d5-5ldn8_4865da86-417f-4703-bcdc-c68b5de3ad59/manager/0.log" Nov 02 15:12:15.484907 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:15.484876 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cp4vx_ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e/dns/0.log" Nov 02 15:12:15.614533 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:15.614497 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cp4vx_ade2f5fa-c7c7-42e8-ad66-80179ef5dd4e/kube-rbac-proxy/0.log" Nov 02 15:12:15.762883 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:15.762807 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 15:12:15.787472 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:15.787441 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5hfrp_b97f6b80-f53b-4fff-97f6-dff42fb5875f/console-operator/2.log" Nov 02 15:12:16.007488 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:16.007460 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zgxlv_087ad47b-bcc1-4332-9706-8f0ed148252d/dns-node-resolver/0.log" Nov 02 15:12:16.953708 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:16.953671 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-c2cbh_36201127-d7ae-4a96-879f-d28da880063c/proxy/0.log" Nov 02 15:12:17.360566 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:17.360481 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2sxls_e9b81de7-b765-4eac-afd2-86d99495af3d/node-ca/0.log" Nov 02 15:12:19.403456 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:19.403426 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8f9c48f5f-jlflv_67f443f5-5750-4236-bcf7-b3239b253c9e/kube-auth-proxy/0.log" Nov 02 15:12:19.545559 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:19.545525 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-259d8_cd410bed-b4ef-467a-95f9-9b520abfd44d/istio-proxy/0.log" Nov 02 15:12:19.819963 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:19.819911 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-8q5hv_1fbf701c-a1aa-4031-8112-4e69e00e103b/istio-proxy/0.log" Nov 02 15:12:19.903110 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:19.903075 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-9pqlk_e3459850-2b2b-4dc0-b1ab-5ffbc87263a6/istio-proxy/0.log" Nov 02 15:12:20.309889 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:20.309863 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-bbc4b5f5d-l54ph_3ce257ea-3be4-412a-92b5-cc93696643c0/payload-pre-processing/0.log" Nov 02 15:12:20.378518 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:20.378488 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-8f7f4697c-89t4w_dc87147a-63b6-4b08-b45a-e90793367a36/payload-processing/0.log" Nov 02 15:12:20.452124 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:20.452097 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7899fcdf65-hl87t_87b5876f-0012-4bdf-b037-25070684476c/router/0.log" Nov 02 15:12:21.097006 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:21.096974 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8f9c48f5f-jlflv_67f443f5-5750-4236-bcf7-b3239b253c9e/kube-auth-proxy/0.log" Nov 02 15:12:21.374819 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:21.374731 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-259d8_cd410bed-b4ef-467a-95f9-9b520abfd44d/istio-proxy/0.log" Nov 02 15:12:21.921706 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:21.921674 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-8q5hv_1fbf701c-a1aa-4031-8112-4e69e00e103b/istio-proxy/0.log" Nov 02 15:12:22.052362 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:22.052327 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-9pqlk_e3459850-2b2b-4dc0-b1ab-5ffbc87263a6/istio-proxy/0.log" Nov 02 15:12:22.849546 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:22.849511 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-bbc4b5f5d-l54ph_3ce257ea-3be4-412a-92b5-cc93696643c0/payload-pre-processing/0.log" Nov 02 15:12:22.980110 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:22.980080 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-8f7f4697c-89t4w_dc87147a-63b6-4b08-b45a-e90793367a36/payload-processing/0.log" Nov 02 15:12:23.116511 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:23.116419 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7899fcdf65-hl87t_87b5876f-0012-4bdf-b037-25070684476c/router/0.log" Nov 02 15:12:23.331798 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:23.331523 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64c547bcd4-s7w4p_a9a48536-da8d-42b1-ac12-d5addcc2cadc/authorino/0.log" Nov 02 15:12:25.686677 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:25.686637 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f9kv5_7120b45c-1c5b-4c7c-863f-1902f96c00f5/serve-healthcheck-canary/0.log" Nov 02 15:12:26.791616 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:26.791573 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-fg7tg_4f856415-9f37-415c-8dd5-65fc1ef08abc/insights-operator/0.log" Nov 02 15:12:26.793483 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:26.793461 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-fg7tg_4f856415-9f37-415c-8dd5-65fc1ef08abc/insights-operator/1.log" Nov 02 15:12:27.773654 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:27.773620 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x6kpv_d32c74f9-dc27-4e9f-a20b-da4c3704acc1/kube-rbac-proxy/0.log" Nov 02 15:12:27.907341 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:27.907309 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x6kpv_d32c74f9-dc27-4e9f-a20b-da4c3704acc1/exporter/0.log" Nov 02 15:12:28.044149 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:28.044061 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x6kpv_d32c74f9-dc27-4e9f-a20b-da4c3704acc1/extractor/0.log" Nov 02 15:12:28.197599 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:28.197564 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-bpfbx-bmlmm_56375571-cada-4ce6-a1b7-54be99ab9904/insights-gathering/0.log" Nov 02 15:12:31.773857 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:31.773825 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-78cdb466dc-dnwzd_2572f605-cbc3-4f28-b54a-ae8b597dbdfc/manager/0.log" Nov 02 15:12:31.904371 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:31.904331 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-78cdb466dc-dnwzd_2572f605-cbc3-4f28-b54a-ae8b597dbdfc/copy-manifests/0.log" Nov 02 15:12:34.055600 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:34.055548 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-67b8558cc4-5qlrn_778bb275-ee0e-4a6f-88a5-774324a3d353/manager/0.log" Nov 02 15:12:35.288400 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:35.288357 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-78cdb466dc-dnwzd_2572f605-cbc3-4f28-b54a-ae8b597dbdfc/manager/0.log" Nov 02 15:12:35.368760 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:35.368726 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-78cdb466dc-dnwzd_2572f605-cbc3-4f28-b54a-ae8b597dbdfc/copy-manifests/0.log" Nov 02 15:12:35.475388 ip-10-0-134-99 kubenswrapper[4866]: I1102 15:12:35.475360 4866 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6cc7b895d5-5ldn8_4865da86-417f-4703-bcdc-c68b5de3ad59/manager/0.log"